AI evidence extraction

LitLLM: A Toolkit for Scientific Literature Review

Shubham Agarwal, Issam Laradji, Laurent Charlin, Christopher Pal · arXiv (Cornell University) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2402.01788

Methodology & findings

Study design

Prototype development with preliminary user validation. The authors conducted a preliminary user study with 5 researchers who tested the system demo to validate efficacy.

Primary method

Design science research with iterative development and preliminary user evaluation

Main result

The system demonstrates substantial reduction in time and effort for literature review compared to traditional methods. The authors note that "Particularly, the users found the 0-shot generation to be more informative about the literature in general while the plan-based generation to be more accessible and tailored for their research paper." Additionally, "our system generated an informative query Multimodal Research: Image-Text Model Interaction and retrieved relevant papers where the top recommended paper was also cited in the original paper."

Research paradigm

Design science / pragmatism

Author conclusions

The authors conclude: "In this work, we introduce and describe LitLLM, a system which can generate literature reviews in a few clicks from an abstract using off-the-shelf LLMs. This LLM-powered toolkit relies on the RAG with a re-ranking strategy to generate a literature review with attribution." They further state: "Given the growing impact of different LLM-based writing assistants, we are optimistic that our system may aid researchers in searching relevant papers and improve the quality of automatically generated related work sections of a paper."

Risk of bias

Small user study sample (n=5) limits generalizability; Selection bias in user selection not described; Limited evaluation of hallucination rates; Potential filtering bias from academic search APIs used; Selection bias: only 5 researchers in preliminary user study with no description of recruitment strategy; Small sample size (n=5) limits generalizability; No control group or comparison to baseline literature review methods; Potential confirmation bias in researcher selection of papers to validate; Limited diversity in user backgrounds not described; Limited user study sample (n=5); potential hallucination risks in LLM generation; dependence on quality of academic search APIs; potential limitations in interdisciplinary coverage

Limitations

The authors explicitly acknowledge: "This work only considered abstracts of the query paper and the retrieved papers, which creates a bottleneck in effective literature review generation." They also note that "we believe that their usage should be disclosed to the readers, and authors should also observe caution in eliminating any possible hallucinations." Additionally, the preliminary study involved only "5 different researchers."

Open questions raised

Authors identify several future directions: (1) exploring academic search through multiple APIs like Google Scholar; (2) expanding from abstract-only processing to ingesting whole papers with efficient PDF parsers for longer context LLMs; (3) building toward more intelligent interactive research assistants.
The authors identify several future directions: exploring academic search through multiple APIs such as Google Scholar, ingesting full papers rather than abstracts with advent of longer context LLMs, leveraging efficient LLM-based PDF parsers, and building intelligent research assistants that could help academics through interactive settings.
Authors identify the need to: (1) explore academic search through multiple APIs such as Google Scholar; (2) ingest whole papers rather than just abstracts using longer context LLMs and efficient LLM-based PDF parsers; (3) develop intelligent research assistants for interactive settings to help academics

Data: Semantic Scholar Open Data Platform (300M+ metadata records); OpenAlex API (scholarly works index); Semantic Scholar Open Data Platform (S2) - Academic Graph and Recommendations APIs; OpenAlex API; Project page with demo: https://litllm.github.io; Demo video: https://youtu.be/E2ggOZBAFw0Code: https://litllm.github.io (project page with demo and toolkit); Project page with toolkit access: https://litllm.github.io; Code snippets provided in Appendix for Semantic Scholar API integration; Project page and toolkit accessible at https://litllm.github.io; demo video available at https://youtu.be/E2ggOZBAFw0Extracted from: pdfAgreement 58%

Explore related topics

Systematic Review Automation Literature Discovery Design Science