AI evidence extraction

PARNESS: A Paper Harness for End-to-End Automated Scientific Research with Dynamic Workflows, Full-Text Indexing, and Cross-Run Knowledge Accumulation

Yuchen Wang, Zhongzhi Luan · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Systems design and implementation of a software framework for autonomous research with modular architecture; evaluation through unit tests, integration tests, knowledge-graph harness testing, and end-to-end pipeline runs on real document collections..

Primary method

Design science / systems engineering. Problem-driven design with five structural limitations (L1–L5) mapped to four design moves (M1–M4) and architectural realizations.

Main result

PARNESS demonstrates that the components of an autonomous research system — crawler, parser, ideator, experimenter, writer, reviewer, knowledge graph, multi-LLM, persistence, monitoring — compose naturally under a thin DAG kernel with a four-field agent contract. "Adding a new pipeline requires no Python; adding a new module requires one class and one registration line. The framework absorbs PaperOrchestra's five-step writing as a single module, AI-Scientist's tree search as a score-gated loop, and InternAgent's eight-agent generation–evolution graph as a sub-DAG."

Research paradigm

Design science / Systems engineering

Author conclusions

"PARNESS demonstrates that the components of an autonomous research system — crawler, parser, ideator, experimenter, writer, reviewer, knowledge graph, multi-LLM, persistence, monitoring — compose naturally under a thin DAG kernel with a four-field agent contract." The authors conclude that "The framework absorbs PaperOrchestra's five-step writing as a single module, AI-Scientist's tree search as a score-gated loop, and InternAgent's eight-agent generation–evolution graph as a sub-DAG" and identify as "The natural next step is a quantitative head-to-head benchmark against PaperOrchestra (writing only), AI-Scientist v2 and InternAgent (full pipeline), with and without the cross-run KG populated."

Risk of bias

No quantitative evaluation against baselines; control-flow validation only, not output quality; potential mode collapse in KG semantic edges; mock LLM behavior may diverge from real LLM behavior in subtle ways.

Limitations

The authors state: "We have not run a head-to-head benchmark against AI-Scientist, InternAgent, PaperOrchestra or autoresearch on a shared task
We have not run human evaluation on generated papers
We have not measured ablations of individual cognitive roles in the ideation layer
Our end-to-end runs validate control flow (data propagates through every stage) and durability (stages do not crash), not quality of the research output." Additionally, "The cognitive-role parallelism is expensive in tokens" and "The Knowledge Graph subsystem's semantic edge step depends on LLM bucketing and we have observed mode collapse on synthetic test data."

Open questions raised

The authors identify the need for: (1) head-to-head benchmarks against existing systems on shared tasks; (2) human evaluation of generated papers; (3) ablation studies of individual cognitive roles; (4) quality metrics for research output beyond control-flow validation; (5) quantification of cross-run accumulation benefits through run_n_vs_n+1 ablations; (6) a fair benchmark fixing research tasks with shared compute and expert-rated evaluation.
"The natural next step is a quantitative head-to-head benchmark against PaperOrchestra (writing only), AI-Scientist v2 and InternAgent (full pipeline), with and without the cross-run KG populated." Human evaluation of generated papers is needed. Ablations of individual cognitive roles in ideation require rigorous measurement. Cross-run accumulation benefit quantification (run_n_vs_n+1 ablation) is a natural follow-up. A shared benchmark task is required for fair comparison.
The authors identify the need for: (1) head-to-head benchmarks against AI-Scientist, InternAgent, PaperOrchestra and autoresearch on shared tasks; (2) human evaluation on generated papers; (3) measured ablations of individual cognitive roles in the ideation layer; (4) quantification of cross-run accumulation benefit through run_n_vs_n+1 ablations; (5) a benchmark fixing research tasks with shared compute and expert-rated evaluation criteria.

Data: PARNESS reference implementation: https://github.com/gtrhythm/PARNESSCode: https://github.com/gtrhythm/PARNESS; https://github.com/gtrhythm/PARNESS (source code and pipeline configurations)Extracted from: pdfAgreement 72%

Explore related topics

AI-Assisted Discovery Multi-Domain Design Science