AI evidence extraction

Peerispect: Claim Verification in Scientific Peer Reviews

Ali Ghorbanpour, Soroush Sadeghian, Alireza Daghighfarsoodeh, Sajad Ebrahimi, Negar Arabzadeh, Seyed Mohammad Hosseini et al. · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1145/3805712.3808368

Methodology & findings

Study design

Artifact development with empirical validation using two complementary benchmarks (controlled and real-world).

Primary method

Design Science with modular, service-oriented architecture emphasizing extensibility and reusability

Main result

The system demonstrates predictable behavior across two complementary benchmarks. On the Controlled Manuscript Claims benchmark, "accuracy on CMC is substantially higher than on RRC across all configurations" with the larger Qwen-2.5-7B model achieving 0.905 accuracy using BM25 retrieval. On real-world review claims, "the dense retriever followed by a reranker provides the most reliable configuration in this setting (ACC 0.287 for Qwen-2.5-7B)." These findings show that "larger verification models (Qwen-2.5-7B) yield noticeably higher accuracy than smaller ones (Qwen-2.5-3B), suggesting that additional capacity helps with nuanced NLI judgments once relevant evidence is available."

Research paradigm

Design science / pragmatist (artifact-driven research with empirical validation)

Author conclusions

The authors conclude that "Peerispect addresses this need by treating reviewer claims as document grounded retrieval and verification queries. The system extracts check worthy claims from reviews, retrieves the most relevant passages from the paper, and verifies them via an NLI style formulation." They further state that "by providing an open, tested, and extensible system for claim level review verification, Peerispect offers a practical playground for the IR community to explore future research on retrieval and inference applied to scholarly communication."

Risk of bias

Selection bias: Only 50 papers sampled from ICLR 2024 submissions, stratified across accepted and rejected papers but limited to single conference; Ground truth annotation bias: Human annotations on RRC were derived from author-reviewer discussions, which may not represent truly objective ground truth; Label ambiguity: Four-label scheme (Supported/Partially Supported/Contradicted/Undetermined) introduces subjective judgment in annotation process; Model-dependent evaluation: Use of GPT-4-mini for claim extraction may introduce biases specific to that model's behavior; Selection bias: Evaluation datasets are limited to ICLR 2024 submissions from OpenReview, which may not represent reviewer patterns across other disciplines, conferences, or journals. The RRC benchmark is manually annotated and subject to annotation bias in label assignment. The use of OpenAI-o4-mini for claim extraction in both benchmarks may introduce systematic bias in what is considered a 'claim.'

Open questions raised

The authors identify that while prior work on factuality assessment and claim verification has focused on general fact-checking or scientific fact verification against large external corpora, there has been little effort to address the unique setting of verifying reviewer statements against a manuscript. They also note that recent benchmarks and models primarily operate at document or discourse level rather than performing grounded, claim-level verification.
The paper identifies the gap of lacking claim-level verification specific to peer review settings. While "prior works on peer-review analysis have largely focused on systemic properties of the review process and high-level assessments of review quality," existing approaches "do not perform the grounded, claim-level verification necessary to identify the specific inconsistencies." The authors position the work as addressing the need for "transparent methods that connect review statements to evidence in the submitted manuscript" as scientific submission volumes continue to grow.
The paper identifies that prior work on peer review analysis has largely focused on systemic properties and high-level assessments rather than claim-level verification. It notes that "most prior work has focused on open domain fact checking or scientific claim verification against large external corpora," whereas "verifying reviewer claims requires grounding statements against a single submitted document, giving way to a distinct and underexplored retrieval problem." The authors position their work as addressing this gap and suggest future directions for the IR community to explore alternatives to their modular pipeline design.

Data: Controlled Manuscript Claims (CMC) benchmark consisting of 500 paper-derived claims extracted from 50 ICLR 2024 manuscripts; Real-World Review Claims (RRC) benchmark comprising 150 manually annotated reviewer claims from 25 papers. All data drawn from publicly available OpenReview entries for ICLR 2024.; Controlled Manuscript Claims (CMC) benchmark: 500 claims from 50 ICLR 2024 manuscripts; Real-World Review Claims (RRC) benchmark: 150 manually annotated reviewer claims from 25 papers; Data sourced from publicly available OpenReview entries for ICLR 2024; Controlled Manuscript Claims (CMC) benchmark: 500 claims extracted from 50 ICLR 2024 manuscripts. Real-World Review Claims (RRC) benchmark: 150 manually annotated reviewer claims from 25 papers. Both derived from publicly available OpenReview entries for ICLR 2024. No explicit statement that datasets will be made available.Code: The paper mentions Docker containerization and service-oriented architecture but does not provide explicit GitHub or GitLab repository URLs. It states the system is designed as a 'reusable research artifact' but availability is not confirmed.Extracted from: pdfAgreement 67%

Explore related topics

AI Peer Review Peer Review Design Science