AI evidence extraction

Knows: Agent-Native Structured Research Representations

Guangsheng Yu, Xu Wang · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed-methods evaluation combining controlled experimentation (140 comprehension questions across 20 papers, 6 LLM agents, 3 capacity tiers), ablation studies (8 conditions), qualitative case studies (3 application scenarios), and deployed system evaluation (community sidecar hub with 10,000+ indexed publications).

Primary method

Design science with iterative specification development, reference implementation, comprehensive evaluation across multiple dimensions, and deployed system validation

Main result

The study found that "sidecar-assisted agents match PDF accuracy with 55% fewer tokens" and that "weak models gain +29 to +42 percentage points" when using the Knows specification. Furthermore, "per-weakness ID references rise from 0% under a free-form PDF prompt to 64-91% under a Knows citation-requesting prompt," demonstrating significant improvements in traceability and efficiency across multiple evaluation dimensions.

Research paradigm

Design science / artifact-centric computing

Author conclusions

The authors conclude: "We presented Knows, a companion sidecar specification for research artifacts that provides agents with structured, schema-validated metadata without modifying the original publication. Evaluation across 20 papers, 14 disciplines, and six LLMs reports three primary findings: (1) weak models gain +29 to +42 percentage points accuracy from sidecar-assisted retrieval, approaching medium-model PDF performance; (2) full sidecars attain accuracy comparable to PDF-reading on strong consumers with 55% fewer tokens, while minimal statement-only sidecars retain 88% of full-sidecar accuracy at 93% token savings; and (3) per-weakness ID references rise from 0% under a free-form PDF prompt to 64-91% under a Knows citation-requesting prompt."

Risk of bias

Circular evaluation bias: all sidecars and questions authored by same LLM (Claude Opus); Uniform template bias: ~7-statement template underestimates optimal granularity for complex papers; Selection bias: only 20 well-known classic papers tested, not modern papers with supplementary materials; Language bias: only English-language papers included; Model selection bias: only 6 models from 3 providers tested; 5 additional models excluded due to insufficient coverage; Limited architectural diversity: no testing of Llama, Gemini, or GPT models; Context truncation: E7 uses only 15K characters per paper due to API limits; Prompt asymmetry in E4: PDF condition prompted for free-form review vs. Knows condition prompted to cite statement IDs; Circular evaluation bias: single LLM (Claude Opus) authored both sidecars and benchmark questions; Template uniformity bias: all sidecars use ~7-statement template regardless of paper complexity; Dataset bias: only 20 well-known classic papers tested, no modern papers with supplementary materials; Model selection bias: five additional models tested but excluded due to insufficient paper coverage; Context truncation bias: E7 limited to first 15K characters per paper; Prompt asymmetry in E4: PDF condition prompted for free-form review while Knows condition prompted to cite statement IDs; Keyword scoring underestimation: acknowledged to underestimate weak-model accuracy by up to 32 percentage points; Circular evaluation bias: all sidecars and benchmark questions authored by Claude Opus; Uniform template bias: E1-E8 used ~7 statements regardless of paper complexity; Selection bias: only well-known classic papers tested, no modern papers with supplementary materials; Scope bias: only six LLM models tested from three providers; five additional models excluded for insufficient coverage; API context limit bias: E7 used only first 15K characters per paper; Prompt asymmetry in E4: PDF condition prompted for free-form review while Knows condition prompted to cite sidecar statement IDs

Limitations

"All sidecars and benchmark questions were authored by the same LLM (Claude Opus), creating circular evaluation bias." Additionally, "The E1-E8 sidecars also use a uniform ∼7-statement template regardless of paper complexity, which E9 shows underestimates optimal granularity for complex papers by up to 57 pp." The authors further note that "All 20 papers are well-known classics
modern papers with supplementary materials, code repositories, and multi-part structures remain untested." Lastly, "Only English-language papers are included" and "E7 uses only the first 15K characters per paper due to API context limits."

Open questions raised

LLM-based semantic verification to complement deterministic lint (currently only structural verification is implemented)
Adversarial semantic-corruption probe for detecting fabricated evidence
Full 840-call matched-output rerun (E10) with numerical accuracy deltas
Calibrated cross-model confidence probe to replace hedging-phrase fallback with threshold τ from Algorithm 1
Live-cycle evaluation of author-native workflows in conference review pipelines
Dataset, model, and benchmark profiles (dataset@1, model@1, benchmark@1)

Data: Benchmark of 140 comprehension questions across 20 papers - available at https://knows.academy/; 140 sidecar records (KnowsRecords) - available at https://knows.academy/; Per-question responses, parsed outputs, token and latency logs, and random seeds - committed to repository at https://knows.academy/; 140 comprehension questions across 20 papers (benchmark dataset); 20 KnowsRecord sidecars for classic papers; Per-question responses, parsed outputs, token and latency logs; Random seeds for reproducibility; All available at: https://knows.academy/ and accompanying open-source repository; https://knows.academy/ - Deployed community sidecar hub with over ten thousand publications; 140 comprehension questions across 20 classic papers - released at https://knows.academy/; Per-question responses, parsed outputs, token and latency logs, and random seeds committed to repositoryCode: https://knows.academy/ - Main project hub containing reference implementation, schema, canonical consumption prompt, benchmark questions, sidecar records, and evaluation harness; Reference implementation at https://knows.academy/ - includes knows-sidecar package (v0.9), knows-gen (scaffold generator), knows-lint (schema linter), and CLI tooling; https://knows.academy/ - Project hub and community sidecar deployment; Open-source repository containing: reference implementation (knows-sidecar package v0.9), schema validator (knows-lint with seven validation checks), scaffold generator (knows-gen), canonical consumption prompt (v1.0 base and v1.1 matched-output), experimental harness, benchmark questions, sidecar records, and evaluation code; https://knows.academy/ - Reference implementation, schema, canonical consumption prompt, benchmark questions, sidecar records, and evaluation harnessExtracted from: pdfAgreement 54%

Explore related topics

AI-Assisted Discovery Scholarly Infrastructure Design Science