AI evidence extraction

ReviewBench: An Extensible Framework for Benchmarking Human and AI Manuscript Review

Natalie N. Khalil, T. J. Reed, Matteo R. Ciccozzi · bioRxiv (Cold Spring Harbor Laboratory) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.64898/2026.04.17.719279

Methodology & findings

Study design

Computational benchmarking framework with systematic analysis of 145,021 review comments from human reviewers, R3, GPT-5.2, and Gemini 3 Pro across three venues (ICLR 2025 n=1,000; Nature Human Behaviour n=142; eLife n=1,000).

Sample

N = 145021, 7 groups

Primary method

Wilcoxon signed-rank tests (non-parametric paired comparisons), Holm-Bonferroni correction for multiple comparisons, Cohen's d for effect sizes, mixed-effects models with Wilcoxon signed-rank tests, Cohen's κ for inter-rater agreement, LLM-based assessment at temperature 0 for consistency, Gemini context cache used to minimize token costs. Software: PostgreSQL (Cloud SQL), Python (implied for processing), OpenReview API v2 for data collection.

Main result

The study found that "AI reviews are more structured than human reviews across all three disciplines, with R3 leading on justification (96.4-97.3%, all p adj < .001, |d| = 1.74 vs. human) and actionability (99.2-99.9%, all p adj < .001, |d| = 1.38 vs. human)." Additionally, "R3 achieves the highest consequential rate across all three venues (87.9-93.2%), compared to 65.5-77.5% for Gemini 3 Pro, 73.7-81.4% for GPT-5.2, and 60.0-67.7% for humans," and crucially, "when restricting to critical comments, human reviewers rank first on consequential rate on more papers than any other source despite a lower average."

Reports effect sizes and confidence intervals.

Research paradigm

Empirical positivism with quantitative benchmarking

Author conclusions

"Together, these data suggest that the most consequential human reviewers outperform AI sources on this metric, but a substantial fraction of reviewers near 0% brings the average down. Combined with systematic differences in critique focus, where humans focus more on contribution and clarity, while AI sources lead on validity, sufficiency, and transparency, these results suggest that AI tools can complement human judgement. AI can reliably scale technical verification, while human judgment remains essential for evaluating contribution and shaping editorial decisions."

Risk of bias

Assessment labels assigned by LLM evaluator rather than human annotators; Rule-based parsing of unstructured human reviews is error-prone; Consequential labels only apply to mapped comments, potentially penalizing sources producing broad editorial comments; Single evaluator model (Gemini 3 Pro) used throughout main analysis; No filtering on paper quality or topic could introduce heterogeneity; LLM evaluator bias: Assessment labels assigned by Gemini 3 Pro; cross-model evaluator study conducted but human annotation would strengthen confidence; Rule-based parsing errors: Unstructured human reviews prone to parsing errors; Scope limitation: Only claim-mapped comments assessed for consequentiality; editorial and broad comments excluded; Venue selection: Limited to venues with publicly available reviews; results may not generalize to other disciplines; Claim extraction: Performed by LLM (Gemini 3 Pro) which may miss or misidentify claims; LLM evaluator bias: Assessment labels assigned by LLM (Gemini 3 Pro), though cross-model evaluator study (n=50) showed comparable rankings between Gemini and GPT-5.2 with no self-favoritism detected; Rule-based parsing errors: Unstructured human review parsing acknowledged as error-prone; Sample representativeness: Nature Human Behaviour sample small (n=142) after filtering; 17 articles excluded due to missing structured remarks; Selection bias: Papers collected from specific venues (ICLR, Nature Human Behaviour, eLife) may not generalize to other disciplines; Methodological confound: Comment volume varies by venue format, affecting interpretation of comment counts; Paper quality proxy: Acceptance decision used as crude control for paper quality, but acknowledged as incomplete

Limitations

"This analysis is limited in that it measures structural and relational properties of reviews, but not their correctness
Assessment labels are also assigned by an LLM evaluator, and although our cross-model evaluator study saw comparable source rankings across evaluators, human annotators would strengthen confidence in the results
Rule-based parsing of unstructured human reviews is also error-prone, further motivating standardized, machine-readable review formats
Lastly, consequential labels apply only to claim-mapped comments, since it is by definition consequential relative to a specific claim."

Open questions raised

Need for more standardized, machine-readable review formats
Human-annotated validation sets required to strengthen confidence in LLM-based assessment
Extension of ReviewBench to other disciplines with publicly available human reviews
As AI review tools mature, ReviewBench can serve as a living benchmark improved by the scientific community
Need for human-annotated validation sets to strengthen confidence in LLM-assigned labels
Standardized, machine-readable review formats required to reduce parsing errors

Data: ReviewBench database hosted on Cloud SQL (PostgreSQL) with read-only public access (Database: reviewbench, User: user, Password: public); ICLR 2025 data retrieved via OpenReview API v2; Nature Human Behaviour data scraped from journal website; eLife data from OpenEvalProject evals repository; ReviewBench framework and database: https://github.com/reviewbench (mentioned as publicly available [24]); PostgreSQL database hosted on Cloud SQL with read-only credentials available: Database: reviewbench, User: user, Password: public; eLife papers and peer reviews from OpenEvalProject evals repository [22]; ICLR 2025 submissions and reviews via OpenReview API v2; Nature Human Behaviour Open Access research articles with linked peer review files; ReviewBench database hosted on Cloud SQL (PostgreSQL), publicly accessible with read-only credentials (Database: reviewbench, User: user, Password: public); Raw data from three venues: ICLR 2025 via OpenReview API v2, Nature Human Behaviour via journal website scraping, eLife from OpenEvalProject evals repositoryCode: ReviewBench publicly available repository (reference [24] in paper, specific URL not provided in text); ReviewBench repository [24] - publicly available; contains raw data fetchers, canonicalization layer, AI review generators, and assessment processor. Operable via CLI.; ReviewBench open-source framework available at [24] (repository URL referenced but specific URL not provided in text)Extracted from: pdfAgreement 53%

Explore related topics

AI Peer Review Peer Review Benchmarking