AI evidence extraction

ReviewBench: An Extensible Framework for Benchmarking Human and AI Manuscript Review

Natalie N. Khalil, T. J. Reed, Matteo R. Ciccozzi · bioRxiv (Cold Spring Harbor Laboratory) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.64898/2026.04.17.719279

Methodology & findings

Study design

Computational benchmarking framework with systematic analysis of 145,021 review comments from human reviewers, R3, GPT-5.2, and Gemini 3 Pro across three venues (ICLR 2025 n=1,000; Nature Human Behaviour n=142; eLife n=1,000).

Sample

N = 145021, 4 groups

Primary method

Wilcoxon signed-rank tests (non-parametric paired comparisons), Holm-Bonferroni correction for multiple comparisons, Cohen's d for effect sizes, mixed-effects models with Wilcoxon signed-rank tests, Cohen's κ for inter-rater agreement, LLM-based assessment at temperature 0 for consistency, Gemini context cache used to minimize token costs. Software: PostgreSQL (Cloud SQL), Python (implied for processing), OpenReview API v2 for data collection.

Main result

The study found that "AI reviews are more structured than human reviews across all three disciplines, with R3 leading on justification (96.4-97.3%, all p adj < .001, |d| = 1.74 vs. human) and actionability (99.2-99.9%, all p adj < .001, |d| = 1.38 vs. human)." Additionally, "R3 achieves the highest consequential rate across all three venues (87.9-93.2%), compared to 65.5-77.5% for Gemini 3 Pro, 73.7-81.4% for GPT-5.2, and 60.0-67.7% for humans," and crucially, "when restricting to critical comments, human reviewers rank first on consequential rate on more papers than any other source despite a lower average."

Reports effect sizes and confidence intervals.

Research paradigm

Empirical positivism with quantitative benchmarking

Author conclusions

"Together, these data suggest that the most consequential human reviewers outperform AI sources on this metric, but a substantial fraction of reviewers near 0% brings the average down. Combined with systematic differences in critique focus, where humans focus more on contribution and clarity, while AI sources lead on validity, sufficiency, and transparency, these results suggest that AI tools can complement human judgement. AI can reliably scale technical verification, while human judgment remains essential for evaluating contribution and shaping editorial decisions."

Risk of bias

Assessment labels assigned by LLM evaluator rather than human annotators; Rule-based parsing of unstructured human reviews is error-prone; Consequential labels only apply to mapped comments, potentially penalizing sources producing broad editorial comments; Single evaluator model (Gemini 3 Pro) used throughout main analysis; No filtering on paper quality or topic could introduce heterogeneity; cross-model evaluator study conducted but human annotation would strengthen confidence; Scope limitation: Only claim-mapped comments assessed for consequentiality; Venue selection: Limited to venues with publicly available reviews; results may not generalize to other disciplines; Claim extraction: Performed by LLM (Gemini 3 Pro) which may miss or misidentify claims; LLM evaluator bias: Assessment labels assigned by LLM (Gemini 3 Pro), though cross-model evaluator study (n=50) showed comparable rankings between Gemini and GPT-5.2 with no self-favoritism detected; Sample representativeness: Nature Human Behaviour sample small (n=142) after filtering; 17 articles excluded due to missing structured remarks; Methodological confound: Comment volume varies by venue format, affecting interpretation of comment counts; Paper quality proxy: Acceptance decision used as crude control for paper quality, but acknowledged as incomplete

Limitations

"This analysis is limited in that it measures structural and relational properties of reviews, but not their correctness
Assessment labels are also assigned by an LLM evaluator, and although our cross-model evaluator study saw comparable source rankings across evaluators, human annotators would strengthen confidence in the results
Rule-based parsing of unstructured human reviews is also error-prone, further motivating standardized, machine-readable review formats
Lastly, consequential labels apply only to claim-mapped comments, since it is by definition consequential relative to a specific claim."

Open questions raised

Need for more standardized, machine-readable review formats
Human-annotated validation sets required to strengthen confidence in LLM-based assessment
Extension of ReviewBench to other disciplines with publicly available human reviews
As AI review tools mature, ReviewBench can serve as a living benchmark improved by the scientific community
Assessment of review correctness, not just structural properties
Future work to improve ReviewBench as a living benchmark with community contributions to track progress in automated manuscript review

Data: ReviewBench database hosted on Cloud SQL (PostgreSQL) with read-only public access (Database: reviewbench, User: user, Password: public); ICLR 2025 data retrieved via OpenReview API v2; Nature Human Behaviour data scraped from journal website; eLife data from OpenEvalProject evals repository; ReviewBench framework and database: https://github.com/reviewbench (mentioned as publicly available [24]); Nature Human Behaviour Open Access research articles with linked peer review filesCode: ReviewBench publicly available repository (reference [24] in paper, specific URL not provided in text); contains raw data fetchers, canonicalization layer, AI review generators, and assessment processor. Operable via CLI.Extracted from: pdf

Explore related topics

AI Peer Review Peer Review Benchmarking