AI evidence extraction

ResearchPilot: A Local-First Multi-Agent System for Literature Synthesis and Related Work Drafting

Peng Zhang · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Systems prototyping with automated and manual end-to-end validation.

Primary method

Design science research with iterative development, functional validation, and systems prototyping

Main result

ResearchPilot demonstrates that literature-review assistance can be implemented as a transparent multi-stage pipeline. The system completed one end-to-end run on the query "What are the recent trends in retrieval-augmented generation for question answering?" in 12.47 seconds, returning 10 papers, 10 structured extractions, a synthesis with 4 consensus items, 2 contradictions, and 3 open gaps, and a related-work draft of 2046 characters. A warning was emitted because Semantic Scholar returned HTTP 429, but the pipeline still completed using arXiv results.

Research paradigm

Design science / Systems engineering

Author conclusions

ResearchPilot demonstrates that literature-review assistance can be implemented as "a transparent multi-stage pipeline instead of a single opaque prompt. By decomposing search, extraction, synthesis, and drafting into explicit modules with typed interfaces, the system becomes easier to inspect, debug, persist, and extend. The main contribution is not a new learning method but a practical open-source architecture for local-first research assistance."

Risk of bias

Selection bias in paper retrieval limited to Semantic Scholar and arXiv; Abstract-only processing may introduce reporting bias; Small evaluation set (3 queries) with limited provider configuration variety; Single-provider testing (Groq) without cross-provider comparative validation; Limited retrieval sources (Semantic Scholar and arXiv only) may introduce selection bias toward certain domains and publication types; Abstract-only processing may bias extraction toward information present in abstracts, excluding important methodological details from full papers; No citation verification mechanism may allow unsupported claims in generated drafts; Small evaluation set (one successful run plus three partial runs) provides limited generalizability evidence

Limitations

ResearchPilot has several important limitations
"First, retrieval is currently limited to Semantic Scholar and arXiv
This is enough for a broad prototype but not for many domain-specific workflows, especially in medicine, engineering, or venues concentrated in closed digital libraries
Second, the system works from abstracts rather than full papers
Abstracts are often insufficient for precise reporting of datasets, experimental settings, or quantitative outcomes
Third, the generated related-work draft is citation-aware but not citation-verified." Additionally, "the current evaluation is functional rather than comparative." Rate limiting from external providers caused failures in repeated requests, demonstrating that "even when the pipeline implementation is stable, end-to-end reliability remains constrained by external model-provider quotas."

Open questions raised

A stronger future study would compare the multi-stage pipeline against simpler baselines, include human ratings for output usefulness and citation quality, and measure stage-level latency and completion rates across a larger evaluation set. Promising future directions include full-text PDF ingestion, domain-specific retrieval connectors, bounded-concurrency extraction, claim or citation verification agents, and reuse of prior report artifacts as retrieval context for new runs.
Full-text PDF ingestion
Domain-specific retrieval connectors
Bounded-concurrency extraction
Claim or citation verification agents
Reuse of prior report artifacts as retrieval context for new runs

Code: ResearchPilot is described as open-source and self-hostable, packaged with Docker Compose, but no explicit GitHub URL is provided in the paper.Extracted from: pdfAgreement 72%

Explore related topics

Systematic Review Automation Literature Discovery Design Science