AI evidence extraction

Towards grounded autonomous research: an end-to-end LLM mini research loop on published computational physics

Haonan Huang · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

3/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Two complementary experimental regimes: (1) Scale mode: autonomous agent deployed on corpus of 111 open-access Quantum ESPRESSO papers filtered from OpenAlex snapshot with reproducibility verification; (2) Depth mode: single agent applied intensively to one carefully chosen Nature Communications paper (Pizzi et al.

Main result

The study found that "across 571 deduplicated quantitative claims, the agent matches 75.8% within 5% and 83.2% within 10% of the published value, with a median deviation of 0.9%." At scale across 111 papers, the agent demonstrated strong reproduction capability, and "∼42% of papers contain at least one substantive methodological concern raised by the agent unprompted." Critically, "eighty-six of eighty-eight critiques (97.7%) emerged only after the agent had actually run a calculation," establishing that "critical scientific scrutiny is execution-bound in an autonomous-agent corpus of this size." In depth, the agent produced a publishable six-page Comment on a Nature Communications paper, with findings that "revise the paper's L G = 5 nm headline conclusion" through independent first-principles calculations.

Research paradigm

Grounded autonomous research: autonomous AI performing real-world science by reading, reproducing, critically evaluating, and extending published work, with every step anchored in the same physical reality the original literature describes.

Author conclusions

The authors conclude that "Closing the mini research loop exercises exactly the three capabilities the introduction identifies as discriminators of real physical science: physics reasoning that cannot be reduced to interpolation (pseudopotential, functional, k-mesh, convergence criterion); multi-scale execution on decades-mature scientific software (QE → Wannier90 → NEGF, with the paper's prose and figures as the only spec); and verifiability against re-runnable physical ground truth at every step." They further state that "Grounding in published physical science is structurally protective against hallucination modes any blank-slate generative system must contend with -not as a feature of our implementation, but as a property of the loop itself." Regarding future directions, they note that "The natural full-loop extension, previewed in §1, is an agent that reads a body of literature, conceives its own question, executes a program of calculations, and writes a publication beyond an incremental Comment."

Risk of bias

Selection bias: Corpus restricted to open-access papers with downloadable supplementary materials, excluding paywalled research; Tool selection bias: Limitation to Quantum ESPRESSO and Wannier90 ecosystem excludes other DFT packages; Scope declaration bias: Agents self-declare in-scope work, potentially introducing confirmation bias; Single model bias: Only Claude Opus 4.6 tested; no comparison with other LLM architectures; Knowledge envelope bias: Five required-reading files constrain agent's knowledge; different envelopes could affect results; Selection bias: Corpus restricted to open-access Quantum ESPRESSO papers, excluding paywalled literature and papers using proprietary or undocumented software; Tool bias: Reproduction success inherently bounded by whether papers use QE/Wannier90 ecosystem; transport solvers and other specialized codes outside this envelope were inaccessible; Knowledge envelope bias: Agent performance artificially enhanced by curation of knowledge files; unknown how performance degrades without this scaffolding; Grading bias: Paper-level grades depend on agent's self-declared scope; out-of-scope claims excluded from grading could mask capability gaps; Single-model bias: Study uses only Claude Opus 4.6; no comparison to other LLM architectures or open-source models; Temporal bias: Corpus drawn from 2010-2024 papers; unknown if findings generalize to contemporary or future literature styles; Detection bias in critique phase: Comparison of agent critiques vs. human peer review uses orthogonal information asymmetry (agent has computational reach without literature fluency; reviewers have converse); not a fair comparison; Prompt convergence bias: Production prompt converged through development batches B1-B4; unknown how sensitive results are to prompt variation; Corpus limitation to open-access Quantum ESPRESSO papers may introduce publication bias; Agent scope declaration bias: agents may under-declare scope to finish within time budget; Tool knowledge envelope bias: knowledge files provided may steer agent toward certain computational approaches; Prompt design bias: boilerplate prompt converged through development batches may encode developer preferences

Limitations

The authors identify multiple harness-level limitations: "Every limitation we encountered traces to the harness, and every one is engineering-addressable along four axes." Knowledge limitations: "Two text files unlock cognitive scope the model already has" but required-reading envelopes are primitive
Tool limitations: the Reproduce stage on Pizzi 2016 required extensive human debugging of legacy NanoTCAD ViDES solver
Resource management: "agents over-subscribe available cores and leak subprocesses across long sessions" and "even when explicitly told to spend unlimited time and reproduce every quantity in the paper, they bias toward narrowing scope and finishing rather than going deep." Visual capability: "Agents miss topological differences between band-structure plots even when asked to compare them, extract numerical values from plots with large error, and never spontaneously look back at their own figures during a pipeline." Additionally, depth mode required human-agent collaboration on tool repair for reasons rooted in legacy software issues.

Open questions raised

Extension to full research loop: reading a body of related literature, conceiving research questions autonomously, and pursuing multi-paper programs
Knowledge harness scalability: moving from five required-reading text files to richer structured-return tool layers
Tool maturity: generalizing beyond legacy software issues (e.g., NanoTCAD ViDES required one week of debugging)
Compute resource management: addressing agent over-subscription of cores and subprocess leaks in long sessions
Planning horizons: improving agent tendency to narrow scope rather than pursue depth despite explicit unlimited-time instructions
Visual and multimodal capabilities: enabling agents to compare band structures, extract values from plots, and iterate on generated figures

Data: All production inputs, outputs, traces, required-reading envelopes, prompts, verdict JSONs, verified Pizzi 2016 reproduction pipeline, COMMENT_FINAL.pdf, cross-machine reproducibility subset, development-batch artifacts, and figure-generation scripts to be released upon publication under CC-BY 4.0 (Methods §M7); All production inputs, outputs, traces, required-reading envelopes, prompts, verdict JSONs, verified Pizzi 2016 reproduction pipeline, COMMENT_FINAL.pdf, cross-machine reproducibility subset, development-batch artifacts, and Pizzi 2016 Reproduce-stage debug journal will be released upon publication under CC-BY 4.0 (per Methods M7). OpenAlex snapshot (March 2026) used for initial corpus construction.; 111 open-access Quantum ESPRESSO papers from OpenAlex snapshot (March 2026); All production inputs, outputs, traces, required-reading envelopes, prompts, verdict JSONs, verified Pizzi 2016 reproduction pipeline, COMMENT_FINAL.pdf, cross-machine reproducibility subset, development-batch artifacts, figure-generation scripts, and Pizzi 2016 Reproduce-stage debug journal (to be released upon publication under CC-BY 4.0)Code: Claude Code CLI (agentic orchestrator with Claude Opus 4.6); Quantum ESPRESSO (open-source DFT software); Wannier90 (open-source Wannier function package); NanoTCAD ViDES (transport solver, instrumented version); Companion QMatSuite work [30] (tool-layer-first design, mentioned but not detailed); Claude Code CLI used as agentic orchestrator (Anthropic proprietary); Quantum ESPRESSO ecosystem (open-source); Wannier90 (open-source); NanoTCAD ViDES (legacy, required human repair). Companion QMatSuite work mentioned as pursuing tool-layer-first design but not detailed. Code availability contingent on publication.; Claude Code CLI (Anthropic); Quantum ESPRESSO repository (cited as [4, 5]); Wannier90 (cited as [8]); NanoTCAD ViDES (cited as [21, 22]); Companion QMatSuite work (referenced as [30])Extracted from: pdfAgreement 44%

Explore related topics

AI-Assisted Discovery Multi-Domain Case Study