AI evidence extraction

Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning

Teo Sušnjak, Peter K. Hwang, Napoleon H. Reyes, Andre L. C. Barczak, Timothy R. McIntosh, Surangika Ranathunga · arXiv (Cornell University) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2404.08680

Methodology & findings

Study design

Case study and empirical validation using a replication approach.

Main result

The study found that "NEFTune emerged as the most effective, with 89.2% of its responses being SUPPORTED, indicative of its strong factual accuracy and alignment with the actual SLR corpus" and that "the finetuned LLM's responses tend to provide a broader and more detailed perspective than ground truth which is often more concise and sometimes more definitive in its assertions." The research demonstrates that domain-specific fine-tuning of LLMs on relatively small SLR datasets can effectively support knowledge synthesis with high fidelity while maintaining source traceability.

Research paradigm

Computational/pragmatist (automation of SLR through computational methods)

Author conclusions

The authors conclude: "This research introduces an SLR-automation framework leveraging finetuned LLMs, presenting a novel and significant methodical advancement in employing AI for academic research. Our comprehensive experiments with LLM finetuning demonstrate that these AI technologies can effectively streamline SLRs, ensuring both efficiency and accuracy in information retrieval. The framework's effectiveness was validated by accurately replicating a pre-existing SLR study, showcasing the practical applicability of our methods." They further state that "the entire proposed framework for automating the final stages of an SLR was validated for viability by replicating an actual published SLR study" and that the work "lays a foundation for further exploration in this area, highlighting the necessity for continuous development of AI tools to enrich and facilitate scholarly research."

Risk of bias

Single SLR case study (learning analytics domain only) - limited generalization; Authors' familiarity with the target SLR paper may introduce selection bias; Domain expert conducting qualitative evaluation may have implicit biases; Reliance on GPT-4 for evaluation and dataset extraction introduces proprietary model bias; Inter-rater reliability issues noted for FEVER metric (correlations 0.49-0.60); Hyperparameter optimization incomplete due to computational constraints; Model selection bias: only Mistral-7B was tested; larger models not evaluated; Dataset size limitation: only 17 papers in target SLR, raising generalizability concerns; Selection bias: Single published SLR chosen as gold standard; Limited model diversity: Only Mistral-7B evaluated; Evaluation bias: GPT-4 used as automated evaluator, which may inherit biases from its training; Small dataset effects: Training on only 17 papers limits generalizability; Question design bias: Researcher-defined questions may reflect author assumptions; Copyright restrictions prevent dataset sharing and external validation

Limitations

The study states: "This study's exploration of LLMs in SLRs was constrained by model selection, focusing on Mistral without assessing larger models over 7 billion parameters that might offer improved outcomes." Additionally, "Dataset access presents a significant limitation, as copyright restrictions by publishing houses impede the public sharing of even summarised content, challenging the reproducibility and validation of our findings." The authors also note "The study's focus on singular-topic questions limits the depth of evaluation regarding the models' ability to handle complex queries involving multiple concepts" and acknowledge that "Performing hyperparameter tuning was particularly challenging given the long training runtimes for model finetuning, and it is highly likely that suboptimal hyperparameters were used in this research as a result."

Open questions raised

Lack of systematic approaches to synthesizing knowledge across multiple papers in SLRs
Limited work on automating the knowledge synthesis phase of SLRs (most efforts focus on retrieval and screening)
Need for methods to mitigate LLM hallucinations in domain-specific tasks
Absence of mechanisms to audit and track source provenance of LLM responses
Limited exploration of fine-tuning LLMs on small datasets representing narrow domains
Need for updated PRISMA reporting guidelines to address AI-assisted SLR processes

Data: Training dataset: 16,423 samples (after permutation augmentation) extracted from 17 academic papers in learning analytics domain; Test dataset: 4,962 data points with 80/20 train-test split; Datasets not publicly available due to copyright restrictions from publishing houses; Initial dataset: 1,503 samples comprised 28% paper-summary-level, 49% paper-level (larger chunks), 31% paper-level (paragraph-level), and 2% SLR-level; Test dataset: 4,962 data points with three permutations per unique question; Note: Authors state copyright restrictions prevent public sharing of summarized content from the 17 papersCode: Python package publicly available at: https://github.com/peterjhwang/slr-helper; Paper references publicly available SLR helper tool for automated Q&A extractionExtracted from: pdf

Explore related topics

Systematic Review Automation Literature Discovery Case Study