AI evidence extraction

Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning

Teo Sušnjak, Peter K. Hwang, Napoleon H. Reyes, Andre L. C. Barczak, Timothy R. McIntosh, Surangika Ranathunga · arXiv (Cornell University) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

3/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2404.08680

Methodology & findings

Study design

Case study and empirical validation using a replication approach.

Main result

The study found that "NEFTune emerged as the most effective, with 89.2% of its responses being SUPPORTED, indicative of its strong factual accuracy and alignment with the actual SLR corpus" and that "the finetuned LLM's responses tend to provide a broader and more detailed perspective than ground truth which is often more concise and sometimes more definitive in its assertions." The research demonstrates that domain-specific fine-tuning of LLMs on relatively small SLR datasets can effectively support knowledge synthesis with high fidelity while maintaining source traceability.

Research paradigm

Computational/pragmatist (automation of SLR through computational methods)

Author conclusions

The authors conclude: "This research introduces an SLR-automation framework leveraging finetuned LLMs, presenting a novel and significant methodical advancement in employing AI for academic research. Our comprehensive experiments with LLM finetuning demonstrate that these AI technologies can effectively streamline SLRs, ensuring both efficiency and accuracy in information retrieval. The framework's effectiveness was validated by accurately replicating a pre-existing SLR study, showcasing the practical applicability of our methods." They further state that "the entire proposed framework for automating the final stages of an SLR was validated for viability by replicating an actual published SLR study" and that the work "lays a foundation for further exploration in this area, highlighting the necessity for continuous development of AI tools to enrich and facilitate scholarly research."

Risk of bias

Single SLR case study (learning analytics domain only) - limited generalization; Authors' familiarity with the target SLR paper may introduce selection bias; Domain expert conducting qualitative evaluation may have implicit biases; Reliance on GPT-4 for evaluation and dataset extraction introduces proprietary model bias; Inter-rater reliability issues noted for FEVER metric (correlations 0.49-0.60); Hyperparameter optimization incomplete due to computational constraints; Model selection bias: only Mistral-7B was tested; larger models not evaluated; Evaluation bias: inter-rater reliability issues, particularly with FEVER metric (H1 correlation 0.49, H2 correlation 0.6 with GPT-4); Hyperparameter tuning bias: authors acknowledge 'it is highly likely that suboptimal hyperparameters were used in this research' due to computational constraints; Dataset size limitation: only 17 papers in target SLR, raising generalizability concerns; Evaluator familiarity bias: authors were familiar with the target SLR study selected for validation; Selection bias: Single published SLR chosen as gold standard; authors' familiarity with the work may introduce bias in validation; Limited model diversity: Only Mistral-7B evaluated; no comparison with larger models; Evaluation bias: GPT-4 used as automated evaluator, which may inherit biases from its training; Inter-rater reliability issues: FEVER metric showed lower correlation (0.49-0.60) between human raters and GPT-4, particularly for 'NOT ENOUGH INFO' category; Small dataset effects: Training on only 17 papers limits generalizability; Question design bias: Researcher-defined questions may reflect author assumptions; Copyright restrictions prevent dataset sharing and external validation

Limitations

The study states: "This study's exploration of LLMs in SLRs was constrained by model selection, focusing on Mistral without assessing larger models over 7 billion parameters that might offer improved outcomes." Additionally, "Dataset access presents a significant limitation, as copyright restrictions by publishing houses impede the public sharing of even summarised content, challenging the reproducibility and validation of our findings." The authors also note "The study's focus on singular-topic questions limits the depth of evaluation regarding the models' ability to handle complex queries involving multiple concepts" and acknowledge that "Performing hyperparameter tuning was particularly challenging given the long training runtimes for model finetuning, and it is highly likely that suboptimal hyperparameters were used in this research as a result."

Open questions raised

Lack of systematic approaches to synthesizing knowledge across multiple papers in SLRs
Limited work on automating the knowledge synthesis phase of SLRs (most efforts focus on retrieval and screening)
Need for methods to mitigate LLM hallucinations in domain-specific tasks
Absence of mechanisms to audit and track source provenance of LLM responses
Limited exploration of fine-tuning LLMs on small datasets representing narrow domains
Need for updated PRISMA reporting guidelines to address AI-assisted SLR processes

Data: Training dataset: 16,423 samples (after permutation augmentation) extracted from 17 academic papers in learning analytics domain; Test dataset: 4,962 data points with 80/20 train-test split; Datasets not publicly available due to copyright restrictions from publishing houses; Training dataset: 16,423 Q&A samples (after permutation) from 17 papers in learning analytics domain; Test dataset: 4,962 data points with same distribution as training dataset; Initial dataset: 1,503 samples comprised 28% paper-summary-level, 49% paper-level (larger chunks), 31% paper-level (paragraph-level), and 2% SLR-level; Training dataset: 1,503 initial samples expanded to 16,423 after permutation (28% paper-summary-level, 49% paper-level from chunks, 31% paper-level from paragraphs, 2% SLR-level); Test dataset: 4,962 data points with three permutations per unique question; Note: Authors state copyright restrictions prevent public sharing of summarized content from the 17 papersCode: Python package publicly available at: https://github.com/peterjhwang/slr-helper; Python package for data curation: https://github.com/peterjhwang/slr-helper; Python package for data curation available at: https://github.com/peterjhwang/slr-helper; Paper references publicly available SLR helper tool for automated Q&A extractionExtracted from: pdfAgreement 46%

Explore related topics

Systematic Review Automation Literature Discovery Case Study