AI evidence extraction

LitLLMs, LLMs for Literature Review: Are we there yet?

Shubham Agarwal, Gaurav Sahu, Abhay Puri, Issam Laradji, Krishnamurthy Dvijotham, Jason Stanley et al. · arXiv (Cornell University) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2412.15249

Methodology & findings

Study design

Multi-stage empirical evaluation combining: (1) automated retrieval experiments using LLM-generated keyword extraction, keyword-based search (Semantic Scholar, Google Search via SERP API), and embedding-based retrieval (SPECTER2); (2) re-ranking strategies including permutation ranking, debate ranking with attribution verification, and SPECTER embedding-based ranking; (3) literature review generation experiments using multiple LLM variants (GPT-3.5-turbo, GPT-4, Llama 2-Chat 7B/13B/70B) under different prompting conditions (0-shot, plan-based, per-citation, sentence-by-sentence); (4) automated evaluation metrics (ROUGE, BERTScore, Llama-3-Eval); (5) human evaluation with 6 expert annotators ranking 160 papers..

Sample

N = 1000, 13 groups

Primary method

t-test for comparing precision and normalized recall with/without attribution verification; McNemar test for measuring statistical significance in human evaluation rankings; Precision and Normalized Recall metrics calculated as: Normalized Recall@k = |Retrieved@k ∩ Ground Truth| / |Retrieved ∩ Ground Truth|; Precision@k = |Retrieved@k ∩ Ground Truth| / k; ROUGE scoring (ROUGE1, ROUGE2, ROUGEL); BERTScore for semantic similarity evaluation; Llama-3-Eval for automated generation quality assessment

Main result

The study found that "combining these ideas improves precision and normalized recall by 10% and 30%, respectively, compared to standard retrieval methods." Additionally, the authors demonstrate that "plan-based prompting and retrieval augmented generation techniques" substantially reduce hallucinations, reporting that "our approach also reduces hallucinations by 18-26%." The plan-based generation approach with GPT-4 achieves ROUGE1 score of 37.198 compared to 33.213 for 0-shot GPT-4, representing a measurable improvement in literature review quality.

Reports effect sizes.

Research paradigm

Empirical-computational with mixed methods (automated evaluation and human assessment)

Author conclusions

The authors conclude: "So, are we there yet? Not quite-but we are getting closer." They state: "Our experiments show that LLMs have significant potential for writing literature reviews, especially when the task is decomposed into these smaller and simpler sub-tasks that are within reach of LLMs, namely through the use of LLM-generated keyword search and embedding-based search for relevant prior work." Furthermore, "Importantly, LLM hallucinations can be substantially reduced using our proposed plan-based prompting and retrieval augmented generation techniques." The authors identify persistent challenges: "1) retrieving all relevant papers consistent with a given human-generated literature review will require new querying strategies; 2) hallucinations can be significantly reduced using plan-based prompting, but our approach does not completely eliminate hallucinations."

Risk of bias

Test set contamination risk with LLMs trained on arXiv data - mitigated by rolling evaluation using papers published after model release dates; Author bias in ground-truth literature reviews - acknowledged as potential source of variability in coverage evaluation; Selection bias in dataset construction - filtered papers requiring 100 retrievable results, potentially excluding niche research areas; Annotator bias in human evaluation - 6 expert annotators, but agreement only on 112/160 examples (70%); Model selection bias - closed-source models (GPT-3.5, GPT-4) may have performance advantages in evaluation; Training data contamination risk in LLMs despite rolling evaluation approach; Author bias in selecting related work citations (acknowledged as different author biases); Limited abstract-only information may not capture full research context; Evaluation metrics (ROUGE) measure token-level similarity but may not account for semantic meaning; Human annotator agreement was 70% (112/160 examples), indicating moderate inter-rater reliability; Test-set contamination risk: Multi-XScience dataset may be in training data of recent LLMs (GPT-4, Llama 2); addressed via RollingEval protocol; Selection bias in ground-truth citations: Authors acknowledge that "authors might have different biases" in citation selection; Low retrieval coverage (7%) limits end-to-end evaluation; authors evaluate components independently; Abstract-only setup may miss relevant papers that would be found using full paper text; Human evaluation agreement was modest (112/160 examples = 70% agreement); Potential researcher degrees of freedom in prompt design and hyperparameter selection

Limitations

The authors state: "Because of the low coverage for retrieval, we evaluate different components independently
During generation, this strategy assumes that we already have filtered relevant papers corresponding to the main paper." They further note that "the retrieval component currently suffers from surface-level information about the papers, while in practice, authors frame search keywords based on information (such as underlying datasets) that might not be present in the abstract." Additionally, "at best, we retrieve just under 7% of the ground truth papers" due to inherent task complexity and the constrained abstract-only setting
The authors acknowledge that "hallucinations can be significantly reduced using our proposed plan-based prompting and retrieval augmented generation techniques" but that "our approach does not completely eliminate hallucinations."

Open questions raised

Improving retrieval coverage beyond 7% through new querying strategies and use of fuller paper content
Completely eliminating hallucinations in LLM-generated literature reviews
End-to-end system evaluation with integrated retrieval and generation components
Better encoding of dataset-specific and methodology-specific information not captured in abstracts
Hyperparameter optimization for open-source model fine-tuning with plan-based generation
Scalability of attribution verification approach with larger candidate pools

Data: RollingEval-Aug dataset: ~1,000 arXiv papers from August 2023 (available via authors); RollingEval-Dec dataset: ~1,000 arXiv papers from December 2023 (available via authors); Extended Multi-XScience corpus with full paper texts (85% of records from S2ORC); S2ORC (Semantic Scholar Open Research Corpus) - used for dataset construction; RollingEval-Aug (August 2023 arXiv papers, ~1,000 papers, ~4.7k documents); RollingEval-Dec (December 2023 arXiv papers, ~1,000 papers); Extended Multi-XScience corpus with full text of research papers (~6.2k rows); S2ORC dataset (Semantic Scholar Open Research Corpus, ~215GB compressed); Available at: http://api.semanticscholar.org/datasets/v1/; RollingEval-Aug: ~1,000 arXiv papers from August 2023 with full cited paper texts (available from S2ORC); RollingEval-Dec: ~1,000 arXiv papers from December 2023 (similar construction); Multi-XScience corpus extended with full text (85% coverage from S2ORC, 215GB compressed); SPECTER2 database: 150M document embeddings of scientific articles; S2ORC (Semantic Scholar Open Research Corpus): 150M papers with full text and citation annotationsCode: https://github.com/LitLLM/litllms-for-literature-review-tmlr; Code released at github.com (as stated in footnote 19); github.com/LitLLM/litllms-for-literature-review-tmlr (mentioned in paper); HuggingFace Transformers and PyTorch implementations documented in Appendix D.2Extracted from: pdfAgreement 49%

Explore related topics

Systematic Review Automation Literature Discovery Experimental