AI evidence extraction

How Far Are We From True Auto-Research?

Zhengxin Zhang, Ning Wang, Sainyam Galhotra, Claire Cardie · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Empirical benchmark study with three complementary evaluation lenses: (1) manuscript-only agentic reviewer (SAR), (2) artifact-aware peer review (PR) where agents inspect code and logs alongside manuscripts, and (3) human meta-review inspection.

Sample

N = 117, 7 groups

Primary method

Descriptive statistics (means, standard deviations) reported for SAR and PR scores. Manual classification and counting of failure modes (fabricated results, underpowered experiments, plan/execution mismatch) with percentage reporting. Domain-specific analysis and breakdown by agent. No formal hypothesis tests or inferential statistics reported.

Main result

The study found that "Claude Code obtains the highest score, outperforms Analemma's FARS, and matches the weighted-average human ICLR 2025 submission" under manuscript-only review, but "manual inspection reveals this picture is overstated: SAR scores are poorly aligned with its actual acceptance decisions and reward plausible framing without verifying experimental substance." Furthermore, "Under artifact-aware PR scores drop sharply, and manual auditing identifies experimental rigor as the major bottleneck, decomposing into three failure modes (fabricated results, underpowered experiments, and plan/execution mismatch) that are highly agent-dependent: Codex 5%/8% paper-vs-artifact mismatch / fabricated references versus Kimi Code 77%/72%, a ∼15× spread."

Reports effect sizes and confidence intervals.

Research paradigm

Empirical-computational (measuring agent-generated research quality through systematic evaluation)

Author conclusions

"In this paper, we systematically investigate the auto-research capabilities and limitations of three frontier agents across 13 CS domains using a minimal scaffold, ResearchArena. We find that experimental rigor is the number-one weakness: agents routinely fail to plan, execute, and faithfully report experiments, limiting both the scope and significance of their papers. Fabricated results and a manuscript-only reviewer that systematically favours honest but narrow framings further raise faithfulness concerns for today's frontier models. In terms of paper quality, all current agents still fall well short of the threshold for top-tier venues. There is still a long way to go for true auto-research."

Risk of bias

Selection bias: Only three frontier agents evaluated; limited generalizability to other model families; Evaluator bias: SAR (automated reviewer) shown to be misaligned with human review standards and overly generous to agent papers; Domain bias: 13 CS domains may not represent full research landscape; Compute platform bias: Different results on A6000 vs H100 hardware; initial experiments on A6000 only; Meta-review bias: Authors serve as meta-reviewers, creating potential conflicts of interest despite focus on objective integrity checks; Selection bias: Only three frontier agents evaluated; results may not generalize to other LLM agents or earlier model versions; Evaluator bias: Authors serve as meta-reviewers, introducing potential for subjective assessment despite focus on objective integrity verification; Domain bias: Limited to 13 computer science domains; findings may not transfer to other research fields; Reviewer calibration bias: SAR scores are poorly aligned with actual human acceptance decisions (0.25 point gap vs. 1.52 for humans), potentially biasing comparisons; Artifact inspection bias: Read-only access prevents complete artifact verification; some issues may be missed; Evaluator bias: Human meta-reviewers (the paper authors themselves) conducted integrity assessment, introducing potential confirmation bias; Limited agent coverage: Only three frontier agents evaluated; results may not generalize to other LLM-based systems; Domain bias: Study limited to computer science; findings may not transfer to other scientific disciplines; Reviewer calibration: SAR (Stanford Agentic Reviewer) shown to be poorly calibrated to human acceptance decisions, with SAR accepting 76% of human-accepted papers but only 41% of Claude Code papers; Selection of research seeds: 13 domains chosen may not be representative of all CS research areas; Artifact integrity assessment: Read-only access restriction prevents verification that reviewers could not modify artifacts, though this was an intentional design choice

Limitations

The authors state "Due to budget constraints, our study evaluates only three agents and therefore does not cover the full space of available age" [text appears truncated]
Key stated limitations include: SAR is poorly calibrated as a sole evaluator of agent-generated papers (it compresses the accept-vs-reject gap to 0.25 points versus human reviewers' 1.52 points)
The focus on integrity rather than novelty in meta-review means novelty assessment remains subjective
The study is limited to 13 computer science domains and does not evaluate other research fields
Compute scaling shows "no consistent improvement" suggesting the agent's experiment design capabilities are the bottleneck, not computational resources.

Open questions raised

Need for principled calibration of agentic reviewers against human review standards
Lack of faithfulness in frontier models on complex end-to-end research tasks
Need for better experiment-planning agents and scaffolds to improve experimental rigor
Inadequate coverage of agent-model families beyond the three tested
How to develop agentic reviewers for agent-generated papers that are better calibrated to human review standards
How to improve agent faithfulness on complex end-to-end tasks beyond single reasoning traces

Data: "we release the full corpus: 117 papers with their code and logs, 351 PR reviews, 117 SAR scores, human inspection results, and the configurable harness."; The authors state "To support the community in tracking progress as models advance, we release the full corpus: 117 papers with their code and logs, 351 PR reviews, 117 SAR scores, human inspection results, and the configurable harness." Specific URLs not provided in the excerpt.Code: The authors indicate the ResearchArena harness and all 117 papers with experimental artifacts will be released, but specific repository URLs are not provided in the paper.; The authors state "To support the community in tracking progress as models advance, we release the full corpus: 117 papers with their code and logs, 351 PR reviews, 117 SAR scores, human inspection results, and the configurable harness." However, no explicit URL is provided in the paper.Extracted from: pdfAgreement 55%

Explore related topics

AI Manuscript Writing Scholarly Infrastructure Mixed Methods