SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation
Weihang Su, Qingyao Ai, Jiaxin Mao, Ziyi Ye, Yiqun Liu · ArXiv.org · 2025
AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.
This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2508.15658
Methodology & findings
Study design
Computational benchmark construction and evaluation framework design.
Main result
The study demonstrates that "even the state-of-the-art systems struggle with the survey generation task, highlighting its difficulty." Specifically, "while our paper retriever makes 36.65% of the ground-truth references available (Recall@100), the best-performing baseline, StepSurvey, only achieves a final Coverage of 6.30%. The simpler RAG baseline is even lower at 2.14%." These findings "underscore that the primary limitation is the generation stage's inability to effectively identify and incorporate relevant information from the provided set of 100 papers."
Research paradigm
Positivist/empiricist (computational benchmarking and evaluation framework)
Author conclusions
The authors conclude: "SurGE is expected to serve as a valuable resource for future research in information retrieval, fostering advancements in both retrieval methodologies and generative AI techniques." They also state that "advanced, multi-stage planning approaches (AutoSurvey and StepSurvey) significantly outperform the standard RAG pipeline in nearly all metrics" but that "all methods still struggle to capture the vast reference space fully." Furthermore: "our results demonstrate that advanced, multi-stage planning approaches (AutoSurvey and StepSurvey) significantly outperform the standard RAG pipeline in nearly all metrics... there remains substantial room for further improvement, particularly in boosting coverage and refining the interplay between local and global survey organizations."
Risk of bias
Selection bias in survey collection: surveys selected only from arXiv (2020-2024), limiting to openly available computer science papers; Citation count bias: minimum citation threshold of 20 may favor certain research areas or publication venues; Annotation bias: annotators were computer science Ph.D. students, potentially introducing domain-specific perspectives; Inter-annotator agreement of 0.792 (Cohen's Kappa) indicates substantial but not perfect agreement, suggesting some subjectivity in quality assessment; Retrieval corpus construction bias: approximately 30% of references unavailable due to closed-access journals, creating incomplete ground truth; Language bias: all surveys and corpus limited to computer science domain in English; Selection bias in survey collection: only papers published between 2020-2024 on arXiv with minimum 20 citations; Annotation bias: selection of Ph.D. students from computer science may not represent diverse perspectives; Corpus bias: 30% of references unavailable, limiting coverage to papers in open-access or arXiv; Evaluation bias: NLI model used for relevance judgment may have inherent biases; Domain bias: benchmark limited to computer science papers only; Selection bias in ground-truth survey collection: focused on arXiv publications (2020-2024) with minimum 20 citations, which may exclude important foundational work and papers from other venues; Annotator bias: Expert annotation performed by only 4 Ph.D. students; potential for disciplinary bias in quality judgments; Evaluation framework bias: NLI-based relevance judgment using cross-encoder/nli-deberta-v3-base may have inherent biases; LLM-as-Judge bias: GPT-4o used for Structure Quality Score and logical coherence evaluation may reflect OpenAI model biases; Corpus incompleteness: 30% of references unavailable, creating selection bias in retrieval pool; Domain restriction: Computer science domain only, limiting generalizability
Limitations
- The authors acknowledge that "Several limitations guide future work, including: (1) The benchmark is limited in the domain of computer science but could be extended to broader domains
- (2) Integration with more advanced retrieval and generation methods could further enhance automated scientific survey generation for computer science." Additionally, the evaluation relies on ground-truth reference sets that "are not assumed to be perfectly complete" and serve "as the best available proxy for expert consensus on a topic's core literature."
Open questions raised
- Benchmark currently limited to computer science domain; extension to other scientific domains needed
- Current retrieval systems achieve only 68.05% recall at k=1000, suggesting need for more sophisticated retrieval paradigms such as LLM-powered search agents
- Generation stage underutilizes retrieved papers: significant gap between retriever performance (36.65% recall@100) and final coverage (best baseline at 6.30%)
- Need for better integration of local (sentence/section-level) vs. global (document-level) planning in survey generation
- Integration of more advanced retrieval and generation methods could enhance automated scientific survey generation
- Extension of benchmark to domains beyond computer science
Explore related topics
Related papers
- PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and ExplanationAndrea C. Tricco · 2018 · 40,391 citations
- Rayyan—a web and mobile app for systematic reviewsMourad Ouzzani · 2016 · 24,664 citations
- Cochrane Handbook for Systematic Reviews of Interventions2019 · 14,420 citations
- Guidance for conducting systematic scoping reviewsMicah D.J. Peters · 2015 · 7,472 citations
- Updated methodological guidance for the conduct of scoping reviewsMicah D.J. Peters · 2020 · 6,688 citations
- Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resourcesMichael Gusenbauer · 2019 · 2,116 citations