OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented LMs
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang et al. · arXiv (Cornell University) · 2024
AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.
This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2411.14199
Methodology & findings
Study design
Computational system development with multi-faceted evaluation combining automated metrics and human expert assessments.
Main result
The study found that "OPENSCHOLAR outperformed both LM-only and retrieval-augmented pipelines, surpassing proprietary and open-source systems. Notably, using fully open-source checkpoints, OPENSCHOLAR outperformed PaperQA2 (Skarlinski et al., 2024), built on proprietary LMs, and production systems like Perplexity Pro, achieving 6% and 10% improvements, respectively." Additionally, "OPENSCHOLAR, when paired with GPT-4o and our trained 8B model, consistently outperformed expert-written responses, with win rates of 70% and 51%, respectively."
Research paradigm
Empiricist/pragmatist (computational systems design and evaluation)
Author conclusions
"In order to further research on LM-based systems that can assist scientific progress, we introduce OPENSCHOLAR and SCHOLARQABENCH, which can help navigate the complex, ever-growing task of scientific literature review. OPENSCHOLAR, a retrieval-augmented system, leverages open-checkpoint LLMs and trained retrieval models to iteratively refine scientific output, addressing challenges such as hallucinations and citation accuracy. SCHOLARQABENCH, a novel large-scale benchmark, provides a standardized way to evaluate literature review automation across multiple scientific domains. In evaluations using SCHOLARQABENCH, OPENSCHOLAR demonstrates substantial improvements, outperforming existing systems, including GPT-4o and the concurrent proprietary system PaperQA2. Our expert evaluation across three scientific disciplines reveals that SCHOLARQABENCH, when paired with fully open-checkpoint models and open-access data stores, generates answers that are more helpful than those produced by expert annotators, who required an hour per annotation. This approach also significantly increases coverage. OPENSCHOLAR using our trained 8B and GPT4o achieves a 51% and 70% win rate against human-generated answers. We opensource the OPENSCHOLAR code, data, model checkpoints, datastores, and SCHOLARQABENCH, along with a public demo, to support and accelerate future research efforts."
Risk of bias
Limited size of human-evaluated dataset (108-110 instances) may introduce statistical variance; Annotator expertise bias - 16 evaluators from specific fields may not capture nuanced differences outside immediate expertise; Potential bias in ScholarQA-CS annotation weights (heuristically determined weight terms combining length, excerpts, rubric items); LLM evaluator biases (Prometheus) - potential self-preference or length bias when evaluating generated outputs; Temporal bias - annotations captured at specific times (July 2024 for SCHOLAR-CS, September 2024 for SCHOLAR-MULTI), may not reflect subsequent developments; Domain coverage bias - ScholarQABench focuses on computer science, biomedicine, and physics; no instances from social sciences or other disciplines; Length confounding factor - OPENSCHOLAR outputs are 2.0-2.4x longer than human-written answers, which may influence human judgment; Selection bias in expert annotator recruitment (Ph.D. holders with publications in specific fields); Small sample size bias in human evaluation (108 questions, 16 annotators); Evaluator expertise bias: annotators may not have deep knowledge across all question domains; Length bias: model outputs 2.0-2.4x longer than human-written answers, potentially influencing human preference judgments; Temporal bias: annotations captured at specific time points, not reflecting evolving literature; Confounding factor: length control experiment shows this may not fully explain preference differences; Inter-annotator agreement limitations (0.68 kappa with ties, 0.70 relaxed); Annotation bias: Annotators had specific expertise areas potentially biasing evaluation toward their fields; Selection bias: Expert annotators (16 Ph.D. students/postdocs) may not represent broader scientific community perspectives; Length bias: Human evaluators noted that OPENSCHOLAR-generated answers were 2.4× longer than human-written answers, potentially influencing preference judgments; Temporal bias: Benchmark annotations captured at specific times (July 2024 for SCHOLAR-CS, September 2024 for SCHOLAR-MULTI) and may not reflect later scientific developments; Domain bias: SCHOLARQABENCH focuses primarily on computer science, biomedicine, and physics with no social sciences or other engineering disciplines; Citation pattern bias: Rubric annotations may have stylistic biases (background, elaborations, challenges) giving LLM-based systems advantages; Datastore bias: Restricted to open-access papers from peS2o, excluding license-protected and proprietary research
Limitations
- The authors state several key limitations: "First, due to the cost and time required to engage expert annotators—individuals with either a Ph.D
- or are currently pursuing one in relevant fields—the evaluation dataset with human-written answers is relatively small (e.g., 110 for CS-LFQA and 108 for expert-written answers)
- This limited dataset may introduce statistical variance and potential biases stemming from the specific expertise of the annotators." Additionally, "our expert annotators identified several limitations
- Despite these issues, we believe OPENSCHOLAR remains a valuable tool for supporting human experts
- First, as highlighted by our expert annotators, OPENSCHOLAR does not consistently retrieve the most representative or relevant papers for certain queries." They also note that "OPENSCHOLAR outputs may contain factual inaccuracies or unsupported information, particularly in versions based on our 8B model, which has limited capacity for instruction-following and scientific knowledge."
Open questions raised
- Development of better retrieval methodologies incorporating citation networks and metadata like publication recency
- Improving training for 8B models to better handle instruction-following and scientific knowledge
- Fair data use in retrieval-augmented LMs, particularly regarding copyright-protected content
- More detailed human analysis on citation accuracy, validity, and factuality
- Expansion of ScholarQABench to additional scientific domains beyond computer science, biomedicine, and physics
- Addressing confounding factors such as output length in human evaluations
Explore related topics
Related papers
- Rayyan—a web and mobile app for systematic reviewsMourad Ouzzani · 2016 · 24,664 citations
- Guidance for conducting systematic scoping reviewsMicah D.J. Peters · 2015 · 7,472 citations
- Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resourcesMichael Gusenbauer · 2019 · 2,116 citations
- Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative AnalysisMikaël Chelli · 2024 · 295 citations
- Artificial intelligence and the conduct of literature reviewsGerit Wagner · 2021 · 275 citations
- Artificial intelligence for literature reviews: opportunities and challengesF. J. Bolaños · 2024 · 188 citations