AI evidence extraction

Knows: Agent-Native Structured Research Representations

Guangsheng Yu, Xu Wang · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed-methods evaluation combining controlled experimentation (140 comprehension questions across 20 papers, 6 LLM agents, 3 capacity tiers), ablation studies (8 conditions), qualitative case studies (3 application scenarios), and deployed system evaluation (community sidecar hub with 10,000+ indexed publications).

Primary method

Design science with iterative specification development, reference implementation, comprehensive evaluation across multiple dimensions, and deployed system validation

Main result

The study found that "sidecar-assisted agents match PDF accuracy with 55% fewer tokens" and that "weak models gain +29 to +42 percentage points" when using the Knows specification. Furthermore, "per-weakness ID references rise from 0% under a free-form PDF prompt to 64-91% under a Knows citation-requesting prompt," demonstrating significant improvements in traceability and efficiency across multiple evaluation dimensions.

Research paradigm

Design science / artifact-centric computing

Author conclusions

The authors conclude: "We presented Knows, a companion sidecar specification for research artifacts that provides agents with structured, schema-validated metadata without modifying the original publication. Evaluation across 20 papers, 14 disciplines, and six LLMs reports three primary findings: (1) weak models gain +29 to +42 percentage points accuracy from sidecar-assisted retrieval, approaching medium-model PDF performance; (2) full sidecars attain accuracy comparable to PDF-reading on strong consumers with 55% fewer tokens, while minimal statement-only sidecars retain 88% of full-sidecar accuracy at 93% token savings; and (3) per-weakness ID references rise from 0% under a free-form PDF prompt to 64-91% under a Knows citation-requesting prompt."

Risk of bias

Circular evaluation bias: all sidecars and questions authored by same LLM (Claude Opus); Uniform template bias: ~7-statement template underestimates optimal granularity for complex papers; Selection bias: only 20 well-known classic papers tested, not modern papers with supplementary materials; Language bias: only English-language papers included; Model selection bias: only 6 models from 3 providers tested; 5 additional models excluded due to insufficient coverage; Limited architectural diversity: no testing of Llama, Gemini, or GPT models; Context truncation: E7 uses only 15K characters per paper due to API limits; Prompt asymmetry in E4: PDF condition prompted for free-form review vs. Knows condition prompted to cite statement IDs; Template uniformity bias: all sidecars use ~7-statement template regardless of paper complexity; Keyword scoring underestimation: acknowledged to underestimate weak-model accuracy by up to 32 percentage points

Limitations

"All sidecars and benchmark questions were authored by the same LLM (Claude Opus), creating circular evaluation bias." Additionally, "The E1-E8 sidecars also use a uniform ∼7-statement template regardless of paper complexity, which E9 shows underestimates optimal granularity for complex papers by up to 57 pp." The authors further note that "All 20 papers are well-known classics
modern papers with supplementary materials, code repositories, and multi-part structures remain untested." Lastly, "Only English-language papers are included" and "E7 uses only the first 15K characters per paper due to API context limits."

Open questions raised

LLM-based semantic verification to complement deterministic lint (currently only structural verification is implemented)
Adversarial semantic-corruption probe for detecting fabricated evidence
Full 840-call matched-output rerun (E10) with numerical accuracy deltas
Calibrated cross-model confidence probe to replace hedging-phrase fallback with threshold τ from Algorithm 1
Live-cycle evaluation of author-native workflows in conference review pipelines
Dataset, model, and benchmark profiles (dataset@1, model@1, benchmark@1)

Data: Benchmark of 140 comprehension questions across 20 papers - available at https://knows.academy/; Per-question responses, parsed outputs, token and latency logs, and random seeds - committed to repository at https://knows.academy/; 20 KnowsRecord sidecars for classic papers; All available at: https://knows.academy/ and accompanying open-source repository; https://knows.academy/ - Deployed community sidecar hub with over ten thousand publicationsCode: https://knows.academy/ - Main project hub containing reference implementation, schema, canonical consumption prompt, benchmark questions, sidecar records, and evaluation harness; Reference implementation at https://knows.academy/ - includes knows-sidecar package (v0.9), knows-gen (scaffold generator), knows-lint (schema linter), and CLI toolingExtracted from: pdf

Explore related topics

AI-Assisted Discovery Scholarly Infrastructure Design Science