AI evidence extraction

AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration

Jiaqi Liu, Shi Qiu, Mairui Li, Bingzhou Li, Haonian Ji, Siwei Han et al. · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Multi-stage empirical evaluation including: (1) experiment-stage benchmarking on 25 ML topics using rubric-assisted LLM judge with three evaluation dimensions (Code Development, Code Execution, Result Analysis weighted 25:25:50), (2) end-to-end ablation across 10 topics and seven human-in-the-loop intervention modes, (3) component ablation isolating each mechanism's contribution using best-of-3 protocol, (4) cross-domain coverage evaluation on 20 scientific-domain tasks (high-energy physics, systems biology, statistics), and (5) case study analysis on Topic T10..

Sample

not–applicable, 7 groups

Primary method

Rubric-assisted LLM judge with deterministic scoring (no inferential statistics). Inter-rater agreement computed on per-leaf basis with re-adjudication protocol for |∆| > 0.20. Best-of-3 rerun protocol used for ablations to account for stochasticity in autonomous agents. Mean and accept-rate aggregation across topics. No hypothesis testing, p-values, or formal statistical inference reported.

Main result

AutoResearchClaw outperforms AI Scientist v2 by a substantial margin. The paper states: "AutoResearchClaw outperforms AI Scientist v2 by 54.7%" on the experiment-stage evaluation, and "AutoResearchClaw (CoPilot) achieves the highest overall strict score (0.648), outperforming AI Scientist v2 (0.419) by 54.7% and AIDEML (0.511) by 26.8%." Additionally, on Result Analysis specifically, "AutoResearchClaw (CoPilot) scores 0.523 against AI Scientist v2's 0.261, a 100.4% relative improvement." The study also demonstrates that "precise, targeted collaboration at high-leverage decision points consistently outperforms both full autonomy and exhaustive step-by-step oversight."

Reports effect sizes.

Research paradigm

Pragmatist/empiricist (computational systems research with quantitative benchmarking)

Author conclusions

The authors conclude: 'AutoResearchClaw outperforms AI Scientist v2 by 54.7%' and position the system as 'a research amplifier that augments rather than replaces human scientific judgment.' They state that 'across seven intervention regimes, we find that targeted intervention at high-leverage decision points consistently outperforms both full autonomy and exhaustive step-by-step oversight.' The authors emphasize that the five mechanisms (multi-agent debate, self-healing execution, verifiable result reporting, human-in-the-loop collaboration, and cross-run evolution) 'interact: past lessons inform debate, debate improves experiment choices, self-healing keeps the pipeline moving, and verification ensures outputs are grounded in actual results.'

Risk of bias

Stochasticity in autonomous research agents - the system uses best-of-N protocol acknowledging path-dependency; LLM backbone choice (GPT-5.3-codex) may bias results toward that model's strengths; Rubric-based evaluation may reflect evaluator assumptions rather than objective research quality; Limited human expertise in domain-specific tasks (physics, biology) may introduce scoring bias; Topic selection bias: topics were selected to be 'CPU-executable in under 10 minutes' which may favor simpler research; Stochasticity in autonomous agents: 'Autonomous research agents are stochastic and path-dependent: a single unlucky branch in hypothesis generation, code repair, or drafting can dominate the final outcome,' mitigated by best-of-3 protocol in ablations but not in main results.; Rubric design bias: Evaluation rubrics are custom-designed by the authors; inter-rater agreement reported but limited cross-validator diversity (two agent reviewers).; Limited baseline comparison: Only compares against AI Scientist v2 and AIDE-ML; other systems excluded from end-to-end comparison.; Benchmark selection bias: 25 ML topics and 20 science-domain topics selected by authors with explicit inclusion criteria; topics are 'CPU-executable in under 10 minutes' which may favor certain research types.; Topic familiarity: Authors and affiliated institutions may have implicit domain knowledge advantages on certain topic classes.; LLM backbone dependency - all systems use same model (GPT-5.3-codex) but results may not generalize to other LLMs; Judge bias - evaluation relies on LLM-based strict judge which may have systematic biases in assessing code quality and result analysis; Selection bias in topic creation - ARC-Bench topics designed to be CPU-executable in <10 minutes may exclude computationally intensive research; Path dependency - stochastic nature of agents means single unlucky branches can dominate outcomes; Citation verification bias - four-layer citation pipeline uses LLM relevance classification which may be error-prone

Limitations

The authors identify several limitations: "The ARC-Bench core (ML01–ML25) is intentionally ML-focused to enable a fair comparison against AIDE-ML and AI Scientist v2" which may limit generalization to other domains
Additionally, "AutoResearchClaw (Full-Auto) fails to produce valid results on 2 topics, both involving complex multi-file implementations with cascading dependencies." The paper also notes that evaluation relies on "an LLM-based relevance check" for citation verification which could introduce model-dependent biases
Furthermore, the component ablation notes that "Autonomous research agents are stochastic and path-dependent: a single unlucky branch in hypothesis generation, code repair, or drafting can dominate the final outcome."

Open questions raised

Existing autonomous research systems model research as linear pipelines rather than iterative cycles
No prior system combines end-to-end execution with multi-agent debate, self-healing, anti-fabrication verification, and cross-run evolution
The optimal level of human intervention in autonomous research remains an open design question
Cross-domain application of autonomous research systems to non-ML domains (physics, biology) remains underdeveloped
Silent semantic failures (e.g., degenerate experimental outcomes that produce identical results across conditions) are not captured by execution metrics alone
Generalisation beyond ML-centric domains: Paper shows preliminary results on HEP, biology, and statistics but notes these remain less mature than ML pipeline.

Data: ARC-Bench benchmark (25 ML topics + 20 scientific domain tasks) - mentioned as open-source, intended for release; ARC-Bench: 25 ML topics (T01–T25) and 20 scientific-domain tasks (P01–P10 high-energy physics, B01–B07 systems biology, S01–S03 statistics). Topics are specified in YAML format with rubrics. Benchmark details in Appendix D.; Pre-cached datasets in sandbox: CIFAR-10, CIFAR-100, FashionMNIST (read-only mounts); sklearn tabular datasets referenced in rubrics (breast_cancer, wine, digits); ARC-Bench benchmark available; datasets include pre-cached CIFAR-10, CIFAR-100, FashionMNIST. Code repository mentioned at: https://github.com/aiming-lab/AutoResearchClawCode: Github: https://github.com/aiming-lab/AutoResearchClaw; https://github.com/aiming-lab/AutoResearchClawExtracted from: pdfAgreement 40%

Explore related topics

AI-Assisted Discovery Research Productivity Experimental