AI evidence extraction

If you really want to know how AI is changing peer review and where it is taking us, talk to early career researchers

David Nicholas, Blanca Rodríguez Bravo, David Clark, Abrizah Abdullah, Jorge Revez, Marzena Swigon et al. · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.33774/coe-2026-vk8w3

Methodology & findings

Study design

Qualitative longitudinal study using semi-structured interviews.

Sample

N = 153, 6 groups

Primary method

Data analysis involved: (1) Reading and approving interview transcripts by interviewees, translating to English where necessary; (2) Transfer to coding sheets structured in tabular form with codification as Yes/No/Uncertain responses; (3) Comparison of H3 and H4 data to determine change; (4) NotebookLM AI tool used for preliminary analysis of interview data with focus on peer review attitudes and AI role. Verification of NotebookLM analysis by national interviewers.

Main result

ECRs are "displaying a complex, shifting attitude toward the integration of AI into peer review, characterized by cautious optimism on the one hand and scepticism on the other." There is "widespread consensus and strong conviction that AI, in its current state, cannot and should not replace the nuanced, expert judgment of human reviewers." The primary change identified is that "AI is now assumed to be an assistive tool (for plagiarism, grammar, citation checks)" with a "shift from replacement to integration and control: focus on ensuring AI remains subservient to human judgment and does not erode professional standards."

Reports effect sizes.

Research paradigm

Interpretive/qualitative

Author conclusions

"ECRs are displaying a complex, shifting attitude toward the integration of AI into peer review, characterized by cautious optimism on the one hand and scepticism on the other. There are widespread consensus and strong conviction that AI, in its current state, cannot and should not replace the nuanced, expert judgment of human reviewers. The attitude remains that AI should be supportive tool and is incapable of evaluating a work's novelty, scientific merit, or ethical implications. There was also a palpable and growing fear that its use will encourage publishers to replace unpaid human labour entirely."

Risk of bias

Convenience sampling - recruitment through publishers and personal networks, not random selection; Unequal cohort composition between H3 and H4 in terms of nationality, discipline, and age; Questions asked in both interview rounds were similar but not identical carbon copies; Only 32 ECRs participated in both rounds, limiting longitudinal comparisons; Potential selection bias toward ECRs with interest in AI and peer review topics; Interviewers were in-country research professors who may have influenced responses; Convenience sampling - not fully representative by age, gender, discipline, and country; Cohort composition differed between H3 and H4 rounds; Only one-third of interviewees participated in both rounds, limiting longitudinal validity; Interview questions were similar but not identical across rounds; Self-reported data on practices and attitudes subject to recall bias; Researcher influence in qualitative interviews; Selection bias: convenience sampling rather than random selection; Sample composition: different demographic makeup between H3 and H4 cohorts; Interview questions: questions asked were similar but not identical between rounds, limiting temporal comparability; Interviewer bias: potential influence from national interviewers conducting interviews; Self-report bias: participants self-reporting practices and attitudes; Small overlap: only 32 of 153+ ECRs participated in both rounds

Limitations

The authors state: "while we sought to be as representative as we could by age, gender, discipline and country practical matter meant we could not always deliver on that and this should be regarded as convenience sample." Additionally, "Change was determined largely by comparing H-3 data with H-4 data
It should be said at the outset, however, that we can only do this in a rough and ready manner
This is because that: a) while the questions asked of both cohorts were very similar, they were not carbon copies
b) the make-up of the cohorts differed in terms of nationality, discipline and age." The authors also note: "This is a preliminary study, part of the long-running, longitudinal Harbingers project, attempting to inform and plan for a further study, which would have a larger and more representative cohort of researchers
Findings should be treated with caution, more as informed observations, filling a big knowledge vacuum."

Open questions raised

The authors identify that "there is just a small literature on the topic of AI, peer review and ECRs partly because the latter are generally a neglected research community." They note their paper "should fill an important gap in our knowledge" about ECR perspectives on peer review. The authors plan a further study "which would have a larger and more representative cohort of researchers."
The authors identify that there is "just a small literature on the topic of AI, peer review and ECRs partly because the latter are generally a neglected research community." They note this paper "should fill an important gap in our knowledge." Future directions mentioned include planning "a further study, which would have a larger and more representative cohort of researchers."
The authors identify that there is "a small literature on the topic of AI, peer review and ECRs partly because the latter are generally a neglected research community" and note their paper "should fill an important gap in our knowledge." They recommend a further study "which would have a larger and more representative cohort of researchers."

Data: Interview data available as part of Harbingers longitudinal study; full interview schedule available on CIBER websiteCode: Not mentionedExtracted from: pdfAgreement 65%

Explore related topics

AI Peer Review Peer Review Qualitative