AI evidence extraction

Ai2 Scholar QA: Organized Literature Synthesis with Attribution

Amanpreet Singh, Joseph Chee Chang, Chloe Anastasiades, Dany Haddad, Aakanksha Naik, Amber Tanaka et al. · ArXiv.org · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2504.10861

Methodology & findings

Study design

Mixed-methods approach: (1) Iterative design and optimization of retrieval, reranking, and generation components using synthetic query evaluation set (N=500 queries, 1,000 passages each); (2) Comparative benchmark evaluation on ScholarQA-CS dataset (100 expert-annotated Computer Science questions) using GPT-4o-as-judge with rubric-based and citation-based metrics (ALCE); (3) Comparative baselines against 7 other systems and 6 LLM-only approaches; (4) Real-world deployment evaluation through user feedback analysis (N=30.2k queries, 2,433 thumbs feedback submissions, 383 free-form feedback instances manually coded); (5) Component optimization experiments (embedding models, quantization methods, ensemble weighting, reranker selection)..

Primary method

Design science research with iterative component optimization; empirical benchmark-driven evaluation; user-centered design (real-world deployment feedback analysis)

Main result

Scholar QA achieves state-of-the-art performance on the ScholarQA-CS benchmark. The system "outperforms several popular LLMs: Llama 3.1 (Dubey et al., 2024), Claude Sonnet 3.7 (Anthropic, 2024) and GPT-4o (Hurst et al., 2024). It even outperforms reasoning models such as Sonnet 3.7 Thinking (Anthropic, 2025), o1-mini (OpenAI, 2024b) and o3mini (Zhang et al., 2025) overall on the Scholar QA-CS benchmark." In real-world deployment, "85% were positive" feedback ratings from 2,433 user submissions.

Research paradigm

pragmatist/design science

Author conclusions

"We present Ai2 Scholar QA, a freely-available long-form literature synthesis system that generates reports for complex scientific questions. We release key components as open source code and public APIs, and report experiments analyzing design decisions and demonstrate state-of-the-art results." The authors conclude that "[scholar QA] achieves state-of-the-art performance on the ScholarQA-CS benchmark" and that real-world deployment demonstrates user value with 85% positive feedback. They note that "follow-up questions may help address insufficient answer detail and cases with a lack of retrieved documents, while improved retrieval may help address incomplete or incorrect references and off-topic responses" as future directions.

Risk of bias

LLM-as-judge evaluation may introduce model bias in assessing response quality; Binary relevance labels generated by GPT-4-turbo with 80% agreement with human annotators—potential disagreement on remaining 20%; Synthetic query generation may not fully represent real user information needs; User feedback collection from self-selected users may skew towards positive responses; Evaluation using LLM-as-judge (GPT-4o) introduces potential bias; noted version dependency where gpt-4o-2024-11-20 inflated scores compared to gpt-4o-2024-08-06; Synthetic query generation method may not fully represent real user information needs; User feedback skewed toward platform users (self-selection bias); Anonymized system responses used for some evaluations may influence rubric annotations; Selection bias in user queries: real-world deployment captured self-selected user base (8,219 unique visitors), not random sample; LLM-as-judge bias: GPT-4-turbo-generated labels only 80% agreement with humans; GPT-4o judge version affects absolute scores; Synthetic query bias: Development set of 500 synthetic queries may not represent distribution of real user queries; Evaluation metric bias: Rubric-based evaluation used anonymized system responses when selecting supporting evidence, inflating scores (noted for Claude 3.0, OpenScholar); Feedback selection bias: Only 2,433 thumbs feedback submitted from 30.2k queries (8.1% response rate); 85% positive could reflect positive response bias; Baseline implementation bias: PaperQA2 had to be substituted with authors' retrieval pipeline for fair comparison due to unreleased corpus; Funding source not explicitly stated but work from Allen Institute (AI2), potentially influencing model and tool selection

Limitations

The paper acknowledges several limitations: (1) "During our initial experiments, we restricted ScholarQA to only summarize the insights conditioned on the quotes extracted from retrieved passages
However, in cases where the retrieved passages were not relevant enough, the system failed to answer the question in favor of just discussing the information in the quotes." (2) Binary relevance labels from GPT-4-turbo achieved only "80% agreement with human annotators on a sample of 100 queries." (3) Schema bias in evaluation: "Scores are inflated because anonymized system responses were used to pick supporting evidence for the rubrics during annotation" (noted in Table 2)
(4) Judge model version variance: "An interesting discovery from our analysis was that with an updated version of GPT-4o (i.e
gpt-4o-2024-11-20) as the judge, the scores are inflated compared to using gpt-4o-2024-08-06." (5) User feedback analysis: 383 instances of "neutral/negative free-form feedback" identified failure modes without quantitative error metrics per category.

Open questions raised

Need for improved handling of cases where retrieved passages are not relevant enough
Requirement for background/introduction sections even when not explicitly requested by users
Follow-up question support to address insufficient answer detail
Improved retrieval to address incomplete or incorrect references and off-topic responses
Potential exploration of reasoning-based models as backbones despite increased latency and cost
Need for improved retrieval to address incomplete or incorrect references and off-topic responses

Data: Semantic Scholar full-text index: 11.7M open-access papers from S2ORC (Lo et al., 2020); Evaluation dataset: 2,426 queries and 225,618 passages (released publicly); ScholarQA-CS benchmark: 100 Computer Science research questions with expert-annotated rubrics (Asai et al., 2024); 11.7M full-text scientific papers index from S2ORC (Lo et al., 2020) via Semantic Scholar APIs; Semantic Scholar index with over 100M abstracts; Evaluation dataset: 2,426 queries and 225,618 passages for re-ranking evaluation; ScholarQA-CS benchmark with 100 expert-annotated computer science questions; Real user queries: 30.2k questions from 8,219 unique visitors over 9 weeks of deployment; Evaluation dataset: 2,426 queries and 225,618 passages with binary relevance labels (released publicly); ScholarQA-CS benchmark: 100 expert-annotated Computer Science research questions (from Asai et al., 2024); S2ORC corpus: 11.7M full-text scientific papers (Lo et al., 2020; indexed in Semantic Scholar API); Semantic Scholar API: 100M+ abstracts and new /snippet/search endpoint with 285.6M passages; Query categorization analysis dataset: 30.2k real user queries from deployment (available via Semantic Scholar)Code: ai2-scholar-qa Python package (open source); Web application code (Typescript and React-based, open sourced); Semantic Scholar APIs and dataset downloads available via Semantic Scholar; Web application source code (Typescript and React-based, open sourced); Public APIs via Semantic Scholar; ai2-scholar-qa: Open source Python package (https://github.com/allenai/scholar-qa or equivalent); Web application: TypeScript/React-based interactive interface (code released as open source); litellm integration: LLM abstraction layer supporting multiple model providersExtracted from: pdfAgreement 51%

Explore related topics

RAG for Research Literature Discovery Design Science