AI evidence extraction

Comparing supervised machine learning and large language models in title-abstract screening

Marco F. Aigner, Matthias Ganzinger, Pascal Probst, Moritz Rinckens, Thomas M. Pausch · Systematic Reviews · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s13643-026-03199-6

Methodology & findings

Study design

Comparative evaluation study assessing performance of both supervised machine learning (SML) and large language models (LLM) in title/abstract-screening.

Sample

N = 40101, 6 groups

Primary method

Bootstrap sampling with 1000 resamples for confidence interval calculation; Delta-method for F1-score confidence intervals (Takahashi et al. method); Confusion matrices calculated using scikit-learn's confusion_matrix function; Train-test split (70% training, 30% testing, stratified by class label); Class weight balancing set to 'balanced' for most SML models (except Naïve Bayes); Random undersampling with scikit-optimize for class imbalance handling; TF-IDF vectorization for SML feature extraction; Lemmatization using WordNetLemmatizer with positional tags; English stopword removal; Percentile-based determination of text length outliers (95% quantile)

Main result

The study found that "both the SML models and the LLM achieve recall values close to or above those of human reviewers." More specifically, "Models have achieved human performance, meaning that they have met or exceeded both a recall of 0.865 and a specificity of 0.79 on the three datasets animal depression (all except for the Naïve Bayes classifier), pancreatic surgery (logistic regression, random forest, support vector machine), and ADHD (llama)." The paper notes that "SML achieves higher F1-scores than the LLM, while the latter is more sensitive towards true positives."

Reports effect sizes and confidence intervals.

Research paradigm

Empirical positivism with comparative benchmarking

Author conclusions

The authors conclude: "Evidence-based clinical practice as well as preclinical research rely on SRs to provide patients with the best possible care according to the latest evidence." They further state that "While both types of models are similarly accessible as they can run on consumer-grade hardware, we consider Llama-3.1-8B-Instruct to be more trustworthy, as it explains its inclusions in natural language which enables human reviewers to validate their correctness." The authors emphasize: "In practice, a model must be accessible, performant, and trustworthy to be used in title/abstract-screening," and conclude that "SML models, on the other hand, are incomprehensible black boxes which are not trivial to implement and interpret for reviewers with only a medical background."

Risk of bias

Human subjectivity in labeling training data (reviewers susceptible to interpretation differences); Potential data leakage: LLM may have been trained on datasets or similar articles in pretraining corpus; Class imbalance in datasets (2.5% to 26.0% inclusion rates); Dataset quality variations (duplicates, corrupted texts, non-English articles); Retrospective dataset use may not reflect prospective performance; Limited hyperparameter tuning could disadvantage certain models; Single prompt approach for LLM evaluation may not represent optimal performance; Potential LLM overfitting to retrospective data due to unknown training corpus composition; Selection bias in datasets: varying class imbalances (2.5% to 26.0% inclusion rates) may not represent typical systematic reviews (3-6% inclusion rates); Measurement bias: human reviewer labels subject to subjectivity and interpretation inconsistencies; Detection bias: automated duplicate detection may miss some duplicates (490 articles with matching titles not manually cross-checked); Data quality issues: noise from OpenAlex data source carries over errors (e.g., author lists instead of abstracts); Missing data: 5.76% of articles without abstracts on average; Model selection bias: minimal hyperparameter tuning may not represent optimal performance; Selection bias: Datasets with varying class imbalances were intentionally selected, which may not represent typical systematic reviews; Data quality issues: Duplicates, corrupted texts, non-English articles, and HTML noise may affect model performance; Human labeling bias: Labels assigned by human reviewers are susceptible to subjectivity and interpretation; LLM training corpus contamination: Datasets may already be part of Llama model's training corpus, leading to potential overfitting; Incomplete data preprocessing: Manual cross-checking was not performed for 490 pancreatic surgery articles with matching titles

Limitations

The authors state: "The results of this study are only partially comparable to those of other studies, as many of these vary in terms of datasets, models and other methods that they used." Additionally, "a limitation inherent to title/abstract-screening itself is that the labels used to train and evaluate the models were assigned by human reviewers, who are susceptible to subjectivity and interpretation." Furthermore, "Because the LLM was pretrained on large webscale corpora, we cannot exclude that some evaluated abstracts, related review texts, or closely similar examples were present in its pretraining data." The study also notes that "Due to the size of the datasets, we relied on automation (filtering non-English abstracts, duplicates) and heuristics (removing lengthy abstracts) to search for noise in the data."

Open questions raised

Need for curating datasets like SYNERGY for reuse in studies on automation of title/abstract-screening
Investigation of how attributes in datasets and inclusion criteria contribute to classification performance
For living systematic reviews, training on earlier update cycles and evaluating on later waves to test temporal generalization under realistic prospective conditions
More elaborate prompting approaches for LLMs such as few-shot examples or in-context learning
Formal prompt-sensitivity analyses for LLM optimization
Systematic comparison of LLMs with different numbers of parameters

Data: Five open-source datasets from SYNERGY (study selection in systematic reviews): ADHD, atypical antipsychotics, calcium channel blockers, oral hypoglycemics, and animal depression datasets; One proprietary dataset: pancreatic surgery data from International Study Group on Pancreatic Surgery (ISGPS) available online at www.EVIglance.com; Titles and abstracts retrieved from OpenAlex index through PyAlex library; All preprocessed datasets and states after each preprocessing step are shared with the manuscript; SYNERGY dataset: open-source repository for study selection in systematic reviews (5 out of 6 datasets); Pancreatic surgery dataset: provided directly by authors; available online through evidence map (www.EVIglance.com); All preprocessing states saved and shared with manuscript; Supplementary data tables and figures provided; Five datasets from SYNERGY repository accessed via OpenAlex index; Pancreatic surgery dataset from International Study Group on Pancreatic Surgery (ISGPS) available at www.EVIglance.com; All six datasets with preprocessing states shared together with manuscriptCode: Python libraries: Pandas, Polars, scikit-learn, Hugging Face, PyAlex, beautifulsoup, natural language toolkit, scikit-optimize, confidenceinterval; Model accessed through: AutoClasses within Hugging Face platform; LLM classification pipeline written in Python scripts; SML classification in Jupyter notebooks; Python libraries: Pandas, Polars, scikit-learn, HuggingFace, Natural Language Toolkit; Code repositories for LLM inference: HuggingFace platform with Llama3.1-8B-Instruct weights; Authors note code can be adapted for local LLM execution using HuggingFace libraries; PyAlex library for OpenAlex access; scikit-learn for SML model implementations; Hugging Face for LLM access; Code for LLM classification pipeline providedExtracted from: pdfAgreement 53%

Explore related topics

Systematic Review Automation Literature Discovery Experimental