AI evidence extraction

ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence

Rui Meng, Bhavana Dalvi Mishra, Jiefeng Chen, Chunliang Li, Palash Goyal, Mihir Parmar et al. · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Comparative empirical evaluation using CoE Integrity Audit applied to 75 papers (15 papers from each of 5 autonomous research systems) across 5 benchmark tasks from ADRS.

Primary method

Design science with iterative refinement. Chain-of-Evidence (CoE) developed as a verifiability standard (analogous to ACID for databases). ScientistOne designed to satisfy CoE by construction through three-stage architecture with native claim provenance. CoE Integrity Audit developed as post-hoc forensic evaluation framework.

Main result

The systematic audit of 75 papers from five autonomous research systems reveals critical evidence chain failures across all baseline systems. "Every baseline system exhibits evidence chain failures: hallucinated references that do not correspond to any real publication (up to 21% of all bibliography entries), method sections that describe algorithms not present in the submitted code, unreproducible scores, and solution code that exploits the evaluator rather than solving the task." ScientistOne achieves zero hallucinated references (0/337 bibliography entries), perfect score verification (12/12), and the highest method-code alignment (14/15), while matching or exceeding human expert solver performance on all five ADRS tasks.

Research paradigm

Design science / engineering with empirical validation

Author conclusions

"Autonomous research systems have reached the point where solver quality alone no longer differentiates them-multiple systems achieve competitive scores on the same benchmarks with vastly different approaches. What separates their outputs is whether the resulting paper can be trusted. Our 75-paper audit shows that no baseline produces papers free of evidence chain failures, and the failures-hallucinated references up to 21%, fictional method sections, scores on the wrong scale-are undetectable by evaluations that only assess surface presentation rather than evidence grounding. Chain-of-Evidence reframes verifiability as a first-class design constraint. ScientistOne demonstrates that an end-to-end pipeline can maintain evidence chains without sacrificing solver competitiveness, and CoE Integrity Audit provides a reusable procedure for auditing any system's output. The gap between ScientistOne's results and the baselines confirms that verifiability is architectural: systems that build evidence chains at claim-production time produce more verifiable outputs than those that reconstruct grounding after the fact."

Risk of bias

Adaptation bias: all baseline systems were adapted to ADRS by authors; original implementations may differ; Infrastructure bias: 16 of 75 runs required retries for infrastructure issues; Model backbone bias: all systems standardized on Gemini 3.1 Pro, potentially favoring certain system architectures; LLM judgment noise: specification violation, reference verification, and method-code alignment rely on LLM judgments with majority voting to reduce noise; Selection bias: only 5 open-source systems included; closed-source systems not evaluated; Domain bias: evaluation focuses exclusively on systems-optimization tasks; generalization to other domains unclear; Selection bias in baseline system adaptation: four open-source systems adapted to ADRS may not represent optimal performance; Fairness concerns: no third-party system was designed for ADRS; adaptation involved judgment calls; LLM judgment noise in I2 and I4 audits: Specification Violation and Method-Code Alignment rely on LLM-judged checks with majority vote; Seed variance: ScientistOne shows high seed variance (e.g., EPLB scores 1, 3, 8 across three seeds); Automation bias: automated review scores (ScholarPeer) may be systematically blind to certain failure modes; Adaptation bias: Four open-source systems were adapted to ADRS; no third-party system was originally designed for ADRS, introducing potential judgment calls in adaptation; Infrastructure bias: 16 of 75 runs required retry due to API timeouts, rate limits, or LaTeX compilation errors, potentially affecting reproducibility differently across systems; LLM judgment noise: Specification Violation and Method-Code Alignment checks rely on LLM-judged majority vote (5 judges); acknowledged noise floor means some violations may be missed or misclassified; Selection bias in baseline choice: Four specific open-source systems were selected; results may not generalize to all autonomous research systems; Automated reviewer bias: ScholarPeer (Gemini-3.1-pro-based) serves as proxy for review quality but is acknowledged to be "systematically blind to certain failure modes"; Hardware dependency bias: EPLB scores excluded from Score Verification because execution time varies with hardware, reducing comparability

Limitations

"We designed CoE and CoE Integrity Audit to be domain-agnostic, but validating that generality requires evaluation across diverse scientific domains
Our current experiments focus on systems-optimization tasks (ADRS), where gold-standard evaluators make score verification and specification violation detection straightforward
Open-ended domains-biology, materials science, theoretical ML-pose harder challenges: evidence chains may involve wet-lab protocols, simulation reproducibility, or proof sketches, each demanding domain-specific verification logic that we have not yet built or tested." Additionally, "Reference Verification checks whether cited references exist-a necessary condition that already catches a surprising number of failures...However, existence is far from sufficient: a real citation can still be used to support a claim the cited paper never made." The audit is limited to structural integrity, not scientific correctness or novelty
"Audit false negatives" exist: "we did not systematically bound false negatives: integrity failures that our checks fail to detect certainly exist, and the true failure rate across all systems is likely higher than reported."

Open questions raised

Extension of CoE to open-ended domains (biology, materials science, theoretical ML) with domain-specific evidence chains
Full reference verification via passage-level natural language inference to verify cited papers actually support claims
Extending verification coverage to qualitative claims beyond numerically falsifiable statements
Verification in settings without deterministic evaluators
Multi-benchmark synthesis and deeper experimental analysis where systems reason about why solutions work
Human expert evaluation replacing automated review proxies

Data: ADRS benchmark (Automated Design of Research Systems): 5 tasks from computer systems (Prism, Cloudcast, EPLB, LLM-SQL, TXN) with fixed evaluators and starter code; MLE-Bench: 5 Kaggle competitions (3D Object Detection, AI4Code, iMet 2020 FGVC7, RSNA Brain Tumor, iNaturalist 2019 FGVC6); Parameter Golf: OpenAI competition (knowledge base of leaderboard solutions up to April 27, 2026, with SOTA score 1.0611); ADRS benchmark (Automated Design of Research Systems) - five systems research tasks: Prism, Cloudcast, EPLB, LLM-SQL, TXN; MLE-Bench Kaggle competitions - five tasks: 3D Object Detection, AI4Code, iMet 2020 FGVC7, RSNA Brain Tumor, iNaturalist 2019 FGVC6; Parameter Golf competition (OpenAI, 2026) - LLM training task with 16MB size and 10-minute training constraints; ADRS benchmark (Automated Design of Research Systems) with five research problems: Prism, Cloudcast, EPLB, LLM-SQL, TXN; MLE-Bench Kaggle competitions (five tasks: 3D Object Detection, AI4Code, iMet 2020 FGVC7, RSNA Brain Tumor, iNaturalist 2019 FGVC6); Parameter Golf competition (OpenAI 2026, with leaderboard solutions up to 2026-04-27 cutoff)Code: All baseline systems are open-source (Sakana AI-Scientist v2, AutoResearchClaw, DeepScientist, AI-Researcher); ScientistOne implementation details provided in appendices but no public repository URL given in paper; Sakana AI-Scientist v2 (open-source, adapted to ADRS); AutoResearchClaw (ARC) (open-source, adapted to ADRS); DeepScientist (open-source, adapted to ADRS); AI-Researcher (open-source, adapted to ADRS); ScientistOne (artifact produced by this paper)Extracted from: pdfAgreement 52%

Explore related topics

Hallucination Control Research Integrity Design Science