AI evidence extraction

ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents

Zhuofeng Li, Yi Lu, Dongfu Jiang, Haoxiang Zhang, Yuyang Bai, Chuan Li et al. · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Comprehensive benchmark-based evaluation combining: (1) Rubric-based evaluation using paper-specific rubrics derived from meta-rubrics, reference reviews, and paper content, assessed by GPT-OSS-120B across 8 dimensions; (2) Numeric-field evaluation measuring rating prediction accuracy (MSE/MAE) and decision prediction (ACC/F1); (3) Component ablation study removing individual agents; (4) Hyperparameter sensitivity analysis; (5) Adversarial robustness testing via attack injection; (6) Human evaluation study with 120 papers rated by experts with average 2,000 Google Scholar citations..

Primary method

Design science research with iterative refinement; computational framework design with ablation studies and empirical validation

Main result

REVIEWGROUNDER consistently outperforms all baseline models across all rubric dimensions. The authors state: "Across all tasks and metrics, REVIEWGROUNDER with a Phi-4-14B-based drafter and a GPT-OSS-120B-based grounding stage consistently outperforms competitive baselines, including AI Scientist, AgentReview, CycleReviewer, and DeepReviewer with the same or even stronger/larger backbones, such as GPT-4o, GPT-4.1, and DeepSeek-R1." Specifically, REVIEWGROUNDER achieves an overall rubric-based score of 10.7699, with 38% improvement over Qwen3-32B, 135% improvement over GPT-4o, and 36% improvement over DeepReviewer-14B. On numeric-field evaluation, REVIEWGROUNDER achieves the lowest rating error (MSE: 1.1607, MAE: 0.8597) and highest decision prediction accuracy (ACC: 0.6809, F1: 0.6699).

Research paradigm

Design Science Research / Mixed-methods empirical evaluation

Author conclusions

The authors conclude: "We present REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that rethinks LLM-based peer review as a staged process of drafting and grounding. REVIEWGROUNDER explicitly decomposes review construction into complementary roles that retrieve relevant literature, analyze paper-specific evidence, and synthesize critiques guided by reviewer guidelines. Together with REVIEWBENCH, a benchmark enabling multi-faceted, rubric-based, and human-aligned evaluation, our work provides an instance-specific evaluation lens and an effective modeling approach for LLM-based peer review. Extensive experiments demonstrate consistent improvements across key review dimensions, with REVIEWGROUNDER outperforming baselines with substantially larger/stronger backbones."

Risk of bias

Dataset composition bias: REVIEWBENCH derived from DeepReview-13K with only 1.3K papers (10% sample), potentially non-representative; Evaluation model bias: GPT-OSS-120B used as evaluator may introduce systematic biases in rubric scoring; Training data leakage risk: Although training data for Drafter is disjoint from REVIEWBENCH, downstream agents use same evaluation model; Human evaluation limited scope: Only 120 papers used in human validation study with 5 expert raters; Venue selection bias: Study limited to ICLR 2024-2025 submissions, may not generalize to other conferences; Retrieval bias: Literature Searcher restricted to papers from 2023 onward, potentially missing important foundational work; Dataset bias: REVIEWBENCH constructed from ICLR 2024-25 submissions only, potentially biased toward computer science venue review standards; Evaluator bias: LLM-based evaluators (GPT-OSS-120B) used for rubric scoring may exhibit inherent biases; Selection bias: Papers filtered for complete PDF content and minimum review count (3+), excluding incomplete submissions; Noise in human reviews: Authors acknowledge that human reviews can be noisy and reviewers do not always follow official guidelines; Dataset composition bias: Papers filtered to those with complete PDF content and ≥3 reviews, potentially excluding problematic submissions; Evaluator bias: GPT-OSS-120B used as fixed evaluator for all rubric-based scoring; Reference review aggregation bias: Uses DeepSeek-R1-Distill-Qwen-32B for aggregation, introducing model-specific biases; Training data leakage risk: Drafter trained on DeepReview-13K subset, though authors claim disjoint from REVIEWBENCH; Literature search bias: Retrieval limited to papers from 2023 onward via Semantic Scholar API; Human evaluation selection bias: Expert raters had average 2,000 Google Scholar citations, may favor certain research styles

Limitations

The authors identify several limitations: "First, due to infrastructure constraints, we do not implement in-the-flow training of the multi-agent workflow (Li et al., 2025), which coordinates agent modules in a trainable, end-to-end manner." Additionally, "Second, our study focuses exclusively on LLM-based reviewers and does not explore higher-level constructs such as LLM-based meta-reviewers or the potential for iterative feedback loops (Thakkar et al., 2025)." Finally, "due to differences in public availability and practical constraints across venues, the cross-venue set we curated is necessarily partial and uneven, limiting the completeness of our framework's performance and robustness evaluation."

Open questions raised

In-the-flow training: End-to-end trainable coordination of agent modules for improved inter-agent coordination
Higher-level constructs: LLM-based meta-reviewers and iterative feedback loops between reviews and feedback
Cross-venue evaluation: More comprehensive and balanced evaluation across different academic venues beyond ICLR
Multi-turn interaction: Exploration of how reviews and feedback could mutually enhance each other through interaction
In-the-flow training of multi-agent workflows for improved inter-agent coordination
LLM-based meta-reviewers and higher-level constructs

Data: DeepReview-13K; REVIEWBENCH; DeepReview-13K (Zhu et al., 2025) - ICLR submissions and reviews from 2024-2025; approximately 12K papers after filtering, with ~1.3K papers sampled for REVIEWBENCH; REVIEWBENCH: curated benchmark with ~1.3K papers, paper-specific rubrics, aggregated reference reviews, and numeric scores/decisions; REVIEWBENCH: Constructed from DeepReview-13K, filtered to ~1.3K papers from ICLR 2024-2025; DeepReview-13K: Used as source, contains ICLR submissions and reviews from 2024-2025Code: REVIEWGROUNDER code; Code availability stated: "The code is available here" (specific URL not provided in text excerpt); Code stated as available but URL not provided in the paper extract: "The code is available here." (placeholder link not specified)Extracted from: pdfAgreement 56%

Explore related topics

AI Peer Review Peer Review Design Science