AI evidence extraction

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

Tingyue Pan, Mingyue Cheng, Daoyu Wang, Yitong Zhou, Jie Ouyang, Qi Liu et al. · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Benchmark construction and evaluation study.

Sample

N = 1111, 4 groups

Primary method

Pearson correlation (r=0.867 between LLM and human annotations); Spearman correlation (ρ=0.867); Quadratic weighted Cohen's kappa (κ=0.866) for inter-rater agreement; Majority voting for aggregating expert annotations; Stratified sampling for human audit (450 pairs stratified by automatic relevance score)

Main result

The study found that "agentic methods outperform single-shot retrievers, but still face limitations in efficiency, constraints-sensitive search, and robustness under different query conditions." Specifically, "the best agentic method improving overall R@100 from 0.214 to 0.314 over the strongest non-agentic baseline, yielding a relative gain of 46.7%." Additionally, "common zero-recall failures are not due to a lack of search effort, but to off-target exploration," with "PaSa and SPAR access hundreds of candidate papers on average across all failed query types."

Reports effect sizes and confidence intervals.

Research paradigm

Empirical-Computational

Author conclusions

The authors conclude: "ScholarQuest offers a transparent and diagnostic testbed for identifying these limitations and tracking future progress in agentic paper search. Future work may develop agents with stronger intent preservation, constraint-aware filtering, and evidence-level reasoning throughout multi-round exploration. We invite the community to use ScholarQuest to study, compare, and build more reliable academic paper search agents."

Risk of bias

Annotator bias in manual query construction (addressed through LLM-based generation); Citation bias in expansion (may miss papers outside well-connected citation networks); Topic seed bias (limited to arXiv CS categories, may not cover full diversity of publication venues); LLM-based relevance adjudication bias (identified in failure analysis with examples of overreliance on topical overlap); Potential annotation bias from LLM-based relevance scoring despite multi-stage adjudication and human audit; Dataset bias: Focus on arXiv computer science papers limits generalizability across disciplines; Title-and-abstract-only relevance judgment may miss fine-grained claims requiring full-text analysis; Potential missed relevant papers from citation expansion depth limitations (up to 2 hops, max 30 citing papers, max 10,000 scored papers per query); Query generation bias: Methodology-guided LLM generation may systematically omit certain query patterns; Query construction bias: Initial topic-to-query mapping relies on LLM (Qwen3-Max) which may introduce systematic biases in how topics are interpreted and converted to research queries; Annotation bias: Multi-stage LLM-based relevance adjudication may exhibit systematic biases; while human audit was conducted, it was stratified on 450 pairs out of the full dataset; Source bias: Multi-source retrieval from Google Search, arXiv, and Semantic Scholar APIs may have different indexing and ranking biases, potentially favoring certain paper types or research communities; Literature environment bias: Reliance on arXiv as the primary literature source excludes conference proceedings, journals not indexed on arXiv, and discipline-specific repositories, potentially biasing toward certain research communities; False-negative bias in answer construction: The paper acknowledges that "automatic construction may still miss some relevant papers in open literature environments"; Evaluation context bias: All evaluated systems interact with the same ScholarBase backend, which may advantage or disadvantage certain retrieval strategies differently than they would perform on other literature environments

Limitations

The paper states: "ScholarQuest is designed as a controlled benchmark for agentic paper search, but it still has several boundaries
First, ScholarQuest focuses on computer science topics and uses an arXiv-grounded literature environment, so it does not cover the full diversity of scholarly communication across disciplines, venues, and publication formats
Second, relevance judgments are based on paper titles, abstracts, and metadata rather than full-text evidence, which makes the benchmark scalable but may miss fine-grained claims that only appear in the body of a paper
Finally, although our answer construction pipeline combines multi-source retrieval, citation expansion, LLM-based relevance adjudication, and human audit, automatic construction may still miss some relevant papers in open literature environments."

Open questions raised

Need for stronger intent preservation in search agents
Need for constraint-aware filtering to handle scope-controlled queries
Need for evidence-level reasoning throughout multi-round exploration
Extensions toward broader literature sources beyond arXiv/CS
Full-text-aware relevance assessment beyond title/abstract/metadata
Stronger answer-set validation to reduce false negatives

Data: ScholarQuest benchmark: 1,111 high-quality queries covering over 1,000 CS topics; ScholarBase: million-scale testbed based on S2 PaperData snapshot with arXiv papers, metadata, and citation relations; ScholarQuest benchmark (1,111 queries with ground-truth answer sets); ScholarBase: million-scale testbed based on arXiv database with paper metadata and citation relations; S2 PaperData snapshot (underlying data source for ScholarBase); ScholarQuest benchmark (to be released); ScholarBase retrieval backend (to be released); S2 PaperData snapshot (Lo et al., 2020) - used as basis for ScholarBaseCode: Released code and retrieval backend ScholarBase mentioned as publicly available, but specific GitHub/GitLab URLs not provided in paper; ScholarBase retrieval backend (mentioned as released but specific URL not provided in text); Evaluation code for nine baseline systems (mentioned as using released inference code); ScholarQuest benchmark repository (announced for release but not yet provided in paper); ScholarBase retrieval backend code (announced for release)Extracted from: pdfAgreement 56%

Explore related topics

AI-Assisted Discovery Literature Discovery Benchmarking