AI evidence extraction

AI And the Editors' Ghost: Who Is the Writer Now?

David Clark, David Nicholas, Abdullah Abrizah, John Akeroyd, Jorge Revez, Blanca Rodríguez‐Bravo et al. · Learned Publishing · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1002/leap.2051

Methodology & findings

Study design

Qualitative case study with thematic analysis.

Sample

N = 60, 2 groups

Primary method

Google NotebookLM was used for generating thematic summaries of interview transcripts. The paper employs qualitative thematic analysis but does not report formal statistical methods, hypothesis testing, or quantitative analysis.

Main result

The study found that "An overarching and thematic summary of the data was produced in minutes, that would otherwise have occupied our research team for weeks." The AI-generated summaries using Google NotebookLM were "immediately plausible and coherent" and "regarded by all national interviewers as impressive," demonstrating the efficiency of AI tools in qualitative data analysis.

Reports effect sizes.

Research paradigm

interpretivist/qualitative

Author conclusions

The authors conclude that this study "is a calibration for our future data analysis," and they reflect that the paper is fundamentally "about how we used AI as an experiment, our reaction to it, how that chimes, resonates, echoes the experiences of the ECRs." This framing suggests their conclusions are exploratory and oriented toward future methodological refinement rather than definitive claims.

Risk of bias

Selection bias from convenience sampling; Small sample size reducing generalizability; Potential confirmation bias in assessing AI-generated summaries; Single AI tool (Google NotebookLM) without comparative analysis against other tools; Convenience sampling - non-random selection of participants; Selection bias - sample limited to ECRs from specific countries; Researcher bias - authors' own impressions and reactions may influence interpretation; Small sample size limiting generalizability; No blinding or independent verification of AI-generated summaries; Convenience sampling; Small sample size; Potential selection bias in ECR recruitment; Subjective impressions of AI output quality by national interviewers

Limitations

The paper uses "a relatively small, convenience sample," which represents a significant methodological limitation in terms of generalizability and representativeness
The authors acknowledge this constraint when they state they "compare the AI generated summaries both against our original data and those first impressions" using this limited sample.

Open questions raised

The authors identify the need for continued calibration and evaluation of AI tools in research data analysis, noting this paper serves as preparation for future data analysis phases of their longitudinal study.
The authors position this work as foundational for future data analysis, suggesting a need for more rigorous evaluation of AI tools in qualitative research. They indicate future phases of the 'Harbingers' project will build upon these initial findings.
The authors identify the need for future data analysis calibration and deeper exploration of how AI tools align with researchers' and early career researchers' experiences in scholarly communication.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 57%

Explore related topics

AI Manuscript Writing Academic Writing Qualitative