AI evidence extraction

AutoSurvey: Large Language Models Can Automatically Write Surveys

Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu et al. · arXiv (Cornell University) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2406.10252

Methodology & findings

Study design

AutoSurvey employs a four-phase methodology: (1) Initial Retrieval and Outline Generation using embedding-based retrieval from 530,000 arXiv papers, with multiple parallel outlines synthesized into a comprehensive outline; (2) Subsection Drafting where specialized LLMs draft each section in parallel using Retrieved-Augmented Generation (RAG) with relevant papers; (3) Integration and Refinement where sections are individually refined and merged; (4) Rigorous Evaluation using Multi-LLM-as-Judge strategy.

Primary method

Design science research with iterative development and comparative evaluation against baselines

Main result

AutoSurvey demonstrates that "at 64k tokens, AutoSurvey achieves 82.25% recall and 77.41% precision in citation quality, outperforming naive RAG-based LLMs (68.79% recall and 61.97% precision) and approaching human performance (86.33% recall and 77.78% precision). In content quality at 64k tokens, AutoSurvey scores 4.73 in coverage, 4.33 in structure, and 4.86 in relevance, closely aligning with human performance (5.00, 4.66, and 5.00 respectively)."

Research paradigm

Design science / Engineering research

Author conclusions

The authors conclude: "In conclusion, to the best of our knowledge, AutoSurvey is the first system to explore the potential of large model agents in writing extensive academic surveys. It proposes evaluation criteria for surveys that align with human preferences, providing a valuable reference for future related research." They further state that AutoSurvey "significantly outperforms naive RAG-based LLM generation and matches human performance in content and citation quality, while also being highly efficient. This advancement offers a scalable and effective solution for synthesizing research literature, providing a valuable tool for researchers in rapidly evolving fields like artificial intelligence."

Risk of bias

Potential bias in human expert evaluation; selection bias in survey topics tested (limited to 20 computer science topics); parametric knowledge bias of LLMs leading to overgeneralization (51% of citation errors); potential variation in evaluation outcomes across different LLM raters.; Selection bias: Human surveys used for evaluation were excluded during retrieval, but the selection criteria for which surveys constitute 'high-quality' benchmarks is not fully transparent; Evaluator bias: LLM-based evaluation relies on model consistency; Spearman's rho of 0.5429 indicates only moderate correlation with human judgment; Parametric knowledge bias: LLMs show heavy reliance on internal knowledge (51% of errors are overgeneralizations), potentially biasing surveys toward patterns in training data; Limited topic diversity: Evaluation restricted to 20 computer science topics related to LLMs; generalizability to other fields uncertain; Annotation bias: Human expert evaluation used for 'meta-evaluation' but no inter-rater reliability (Cohen's kappa, Fleiss' kappa) reported; Corpus bias: Retrieval database of 530,000 papers limited to arXiv computer science papers; coverage of other fields/databases not addressed; Evaluation bias: Multiple LLM evaluators may share similar biases in assessing survey quality; Selection bias: Human surveys used for comparison were excluded from retrieval database, but may not represent typical human-written surveys; Model bias: AutoSurvey relies on parametric knowledge of LLMs, leading to overgeneralization (51% of errors); Citation hallucination: LLMs may generate plausible but inaccurate citations; Database bias: Evaluation limited to arXiv computer science papers (530,000 papers)

Limitations

The authors identify three main categories of citation errors in their manual analysis: "Misalignment occurs when the connection between them is incorrectly made, such as an irrelevant citation
Misinterpretation happens when the claim and source are related, but the claim incorrectly represents the information from the source
Overgeneralization occurs when a claim extends the conclusions of the source material to a broader context than is supported." Notably, "Among the three types of errors, overgeneralization accounts for the largest proportion (51%), indicating that LLMs still rely heavily on their parametric knowledge for writing." Additionally, the authors note that "as our method relies on the performance of large models, it inevitably contains citation errors
Therefore, the generated survey content is intended for reference only."

Open questions raised

The paper identifies the lack of comprehensive surveys in many fields and the challenge of rapid information accumulation in fields like AI that outpaces scholarly review capacity. The authors note the need for more efficient methods to synthesize expanding literature and propose their system as addressing these gaps.
Limited evaluation scope: Evaluation restricted to 20 topics in computer science; generalizability to other academic domains unexplored
Moderate LLM-human alignment: Spearman's rho of 0.5429 indicates room for improvement in automated evaluation consistency with human judgment
Overgeneralization persistence: 51% of citation errors are overgeneralizations, indicating need for improved mechanisms to constrain LLM parametric knowledge
Long-form generation challenges: While addressed for surveys up to 64k tokens, scalability beyond this length and to other long-form formats not explored
Real-time knowledge currency: RAG mechanism relies on arXiv database; timeliness and completeness of retrieval for emerging topics not quantified

Data: 530,000 computer science papers from arXiv used as retrieval database; human expert evaluations on 20 survey topics; example survey on Emotion Recognition provided in Appendix F; 530,000 computer science papers from arXiv (retrieval database; not publicly released as a specific dataset); 20 survey topics used for evaluation (Appendix references Table 6 but specific topics/papers not enumerated in provided text); 100 unsupported claims manually analyzed (error categorization detailed in limitations section); ArXiv computer science papers corpus (530,000 papers used for retrieval); Example survey on Emotion Recognition provided in Appendix FCode: Not mentioned in paperExtracted from: pdfAgreement 59%

Explore related topics

Systematic Review Automation Literature Discovery Design Science