AI evidence extraction

AI in the gatekeeper’s chair: elite researchers’ perceptions of AI-assisted feedback in journal peer review

Heng Li · Assessment & Evaluation in Higher Education · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1080/02602938.2026.2691499

Methodology & findings

Study design

Sequential mixed-methods approach combining a randomised experimental survey (N = 495) with qualitative in-depth interviews (n = 47) among elite researchers

Sample

N = 542, 2 groups

Main result

The study found that "AI-assisted reviews were viewed as deficient in fairness, usefulness, and acceptance compared to human-led evaluations." Additionally, qualitative evidence identified "a dual-layered aversion to AI-assisted feedback, concerning both the technology (AI use) and the agent (AI user)," with scholars perceiving "AI-generated critiques as devoid of the necessary disciplinary nuance required for high-stakes evaluation."

Reports effect sizes.

Research paradigm

Mixed-methods (sequential pragmatism combining quantitative and qualitative approaches)

Author conclusions

The authors conclude that "the integration of AI into peer review may erode trust in the research evaluation process and journals should implement robust governance that preserves human oversight."

Risk of bias

Selection bias: Study limited to elite researchers (Nature and Science authors only), not representative of broader scientific community; Potential social desirability bias in interview responses; Randomization procedure not described for experimental survey; No mention of blinding procedures; Selection bias: Study limited to elite researchers (Nature and Science authors) - may not represent broader research community; Social desirability bias: Researchers may report negative views of AI to conform to perceived peer expectations; Interviewer bias: 47 in-depth interviews subject to potential interviewer effects; Framing effects: Perception of AI-assisted feedback may be influenced by how the intervention was presented; Selection bias: Cohort limited to elite researchers (Nature and Science authors), which may not represent broader scientific community; Potential response bias in survey and interview participation; Possible experimenter/investigator bias in qualitative interview interpretation

Open questions raised

The abstract indicates that "the psychological impact of this transition on the scientific community remains only partially understood," suggesting this as a research gap being addressed.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 68%

Explore related topics

AI Peer Review Peer Review Mixed Methods