AI evidence extraction

Comparing Single-Agent and Multi-Agent Strategies in LLM-Based Title-Abstract Screening

Irina Radeva, Teodora Noncheva, Lyubka Doukovska, Иван Попчев · Electronics · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.3390/electronics15081661

Methodology & findings

Study design

Empirical comparative study using structured evaluation framework.

Sample

N = 200, 11 groups

Primary method

Cohen's Kappa (κ) and Prevalence-Adjusted Bias-Adjusted Kappa (PABAK) for inter-rater reliability. Binary classification metrics: Recall (sensitivity), Precision (positive predictive value), F1 Score (harmonic mean), and Work Saved over Sampling at 95% recall (WSS@95). Wilson score method for 95% confidence intervals on proportions. McNemar's exact test for paired binary outcome comparisons, using binomial distribution under null hypothesis of equal classifier performance. Exact two-sided p-values computed. Statistical power estimated using normal approximation to McNemar statistic. Ratcliff/Obershelp algorithm (difflib.SequenceMatcher, Python 3.11.14) for title similarity deduplication (threshold ≥0.85). Software: Python 3.11.14, MongoDB v.5.0.23, FastAPI v.0.100.1, Apple MLX framework (mlx v.0.30.3, mlx-metal v.0.30.3, mlx-lm v.0.30.4) for local 4-bit quantised LLM inference.

Main result

The study found that "model selection was the primary determinant of screening performance, outweighing strategy selection. The single-agent strategy with Qwen 2.5 7B in few-shot mode achieved the highest point estimates among all configurations (recall = 100.0%, precision = 70.4%, F1 = 82.6%, WSS@95 = 43.4%)." Additionally, "Pairwise McNemar's exact tests confirmed that S1 was statistically superior to all four multi-agent alternatives (p ≤ 0.0005, power > 0.93)" and "Confidence-weighted aggregation (S4) produced results identical to majority voting (S2), indicating that self-reported confidence from 7-8B parameter models does not provide additional discriminative value in this setting."

Reports effect sizes and confidence intervals.

Research paradigm

positivist/empiricist

Author conclusions

"Three principal findings emerged, each addressing one of the research questions posed in Section 1.3. RQ1 (coordination strategy effect): Model selection was the primary determinant of screening performance, outweighing strategy selection. The single-agent strategy with Qwen 2.5 7B in few-shot mode achieved the highest point estimates among all configurations (recall = 100.0%, precision = 70.4%, F1 = 82.6%, WSS@95 = 43.4%). Pairwise McNemar's exact tests (Section 4.6) confirmed that S1 was statistically superior to all four multi-agent alternatives (p ≤ 0.0005, power > 0.93), while no significant differences were detected among the remaining configurations (all p = 1.0, power ≤ 0.07)." The authors conclude that "Coordination overhead appears to be a general limitation at the 7-8B scale" and note that "The blockchain-based audit mechanism combining private chain logging, Open-Timestamps anchoring, and Zenodo archival provides decision-level traceability that addresses part of the documented reproducibility gap in AI-aided screening."

Risk of bias

Selection bias: Gold Standard drawn only from keyword-rich papers (Pool A), intentionally overrepresenting terminologically ambiguous cases, limiting generalisability to full corpus; Calibration data leakage: Few-shot examples selected from error patterns observed on the same Gold Standard, despite exclusion of those 10 papers from evaluation; Moderate inter-rater agreement (κ = 0.515) introduces label noise and uncertainty in ground truth, particularly affecting precision estimates on disputed subset (precision 0.375-0.452 vs. 0.707-0.821 on agreed subset); Limited model diversity: All four models are instruction-tuned transformers in 7-8B parameter range; architectural homogeneity may suppress multi-agent benefits; Single prompt template: No alternative wordings or instruction structures tested; results may be sensitive to prompt formulation; Single-domain validation: Findings specific to blockchain e-voting domain with terminological overlap; generalisability to other domains unvalidated; Small evaluation set: Gold Standard of 200 papers (190 for few-shot evaluation) limits statistical power for precision comparisons; Selection bias: Gold Standard sample intentionally over-represents keyword-rich papers with borderline cases rather than statistically representative sample; Label uncertainty: Moderate inter-rater agreement (Cohen's κ = 0.515) introduces noise into ground truth labels; Data leakage: Few-shot calibration examples drawn from same corpus, constituting indirect data leakage despite exclusion from evaluation; Limited model diversity: All tested models were 7-8B instruction-tuned transformers, limiting architectural diversity; Domain-specific effects: Results derived from terminologically complex blockchain-e-voting domain may not generalize; Selection bias: Gold Standard constructed from keyword-enriched subset (Pool A), intentionally over-representing terminologically ambiguous cases rather than statistically representative sampling; Label noise: Moderate inter-rater agreement (Cohen's κ = 0.515) introduces uncertainty in ground truth, particularly evident in disputed subset where precision dropped to 0.375-0.452; Data leakage: Few-shot calibration examples selected from zero-shot error patterns on same corpus, constituting indirect data leakage despite exclusion of 10 calibration papers from evaluation; Model selection bias: Four models all from instruction-tuned transformer 7-8B families with limited architectural diversity, potentially affecting conclusions about strategy effectiveness; Prompt bias: Single prompt template used for all experiments without testing alternative wordings or instruction structures

Limitations

"The framework was validated on a single domain
The results should therefore be interpreted as domain-specific, and generalisability to other domains remains to be established
The Gold Standard was constructed exclusively from Pool A, papers containing voting-related keywords, which intentionally over-represents terminologically ambiguous cases..
The Gold Standard of 200 papers (190 for evaluation) limits the statistical power of precision comparisons..
All models tested were instruction-tuned transformer variants in the 7-8B parameter range, deployed on local hardware
This limited architectural diversity may have contributed to the observed S4-S2 equivalence..

Open questions raised

Generalisability across domains: Results derived from single blockchain e-voting domain; validation needed in domains with different terminological characteristics
Scaling to larger models: Whether larger models (13B-70B parameters) benefit from multi-agent coordination remains unexamined; recent evidence at higher parameter scales suggests this possibility
Prompt sensitivity: Systematic exploration of alternative prompt structures and output formats not conducted; single template used throughout
Full-text integration: Retrieval-augmented generation (RAG) could provide full-text access during screening to improve precision
Active learning integration: Iterative refinement could select most informative papers for human review to improve decision boundaries
Domain-specific criteria derivation: How best to systematically derive domain-specific inclusion/exclusion criteria; sensitivity analysis of individual criteria needed

Data: Blockchain audit trail deposited on Zenodo with persistent DOI (Tier 3 verification mechanism); Corpus of 2036 papers (4021 initial, deduplicated to 4021 unique, filtered to 2036); Gold Standard of 200 papers (190 used for evaluation after calibration exclusion); Project data deposit available in GitHub repository: https://github.com/scpdxtest/PaSSER-SR; Corpus of 2,036 blockchain-based e-voting papers (available through PaSSER-SR platform); Gold Standard of 200 papers with dual reviewer annotations; Project repository: https://github.com/scpdxtest/PaSSER-SR; Project repository with code and data; Blockchain audit trail deposited on Zenodo with persistent DOICode: https://github.com/scpdxtest/PaSSER-SR - main project repository containing PaSSER-SR platform (web-based system for screening workflow), all experimental configurations, prompts, evaluation procedures, and analysis scripts; https://github.com/scpdxtest/PaSSER-SR (PaSSER-SR platform with all experimental code, prompts, and evaluation procedures); PaSSER-SRExtracted from: pdfAgreement 53%

Explore related topics

Systematic Review Automation Literature Discovery Experimental