AI evidence extraction

Do We Need Bigger Models for Science? Task-Aware Retrieval with Small Language Models

Florian Kelber, Matthias Jobst, Yuni Susanti, Michael Färber · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Empirical evaluation study combining system design, implementation, and benchmarking across three scholarly datasets.

Sample

N = 108, 7 groups

Primary method

Study primarily uses descriptive comparison of model performance across configurations. Evaluation metrics include: (1) LLM-based judges (Prometheus models) scoring on 1-5 Likert scales for multi-paper QA, (2) Accuracy and F1 scores for single-paper QA, (3) ROUGE-1, ROUGE-2, ROUGE-L for text compression, (4) BERTScore for semantic similarity. No formal statistical significance testing (t-tests, ANOVA, etc.) reported.

Main result

The study found that "small instruction-tuned models can approach the performance of larger systems when paired with appropriate retrieval strategies, particularly for tasks requiring grounded, citation-based answers." Additionally, "retrieval and model scale are complementary rather than interchangeable," with findings demonstrating that while retrieval design can partially compensate for smaller models, "model capacity remains important for complex reasoning tasks."

Reports effect sizes.

Research paradigm

Empirical (computational experiments and benchmarking)

Author conclusions

"Our evaluation results show that small instruction-tuned models can approach the performance of larger systems when paired with appropriate retrieval strategies, particularly for tasks requiring grounded, citation-based answers." However, the authors conclude that "retrieval and model scale should be viewed as complementary components rather than interchangeable solutions." They state: "Overall, our study suggests that progress in retrieval quality is as critical as progress in model scaling for building practical and efficient scholarly assistants."

Risk of bias

Domain mismatch between training corpus (unarXive, primarily computer science and physics) and evaluation dataset (PubMedQA, biomedical domain); Publication bias acknowledged in underlying literature; Domain skew in scholarly literature may influence system behavior; Selection of specific evaluation datasets may not represent full range of scholarly tasks; Domain mismatch bias: Evaluation on PubMedQA (biomedical domain) using corpus primarily from computer science and physics (unarXive); Publication bias: Authors acknowledge "potential biases in the underlying literature (e.g., publication bias, domain skew) that may influence system behavior"; Retrieval quality bias: Dense vector search may return only loosely related passages; Task-specific fine-tuning specialization: Fine-tuned models show reduced general-domain recall compared to pretrained counterparts; Domain mismatch bias: PubMedQA evaluation uses unarXive corpus (primarily computer science and physics) causing performance degradation in biomedical domain; Corpus coverage bias: 165K-paper unarXive subset may not represent full scholarly literature; Selection bias in evaluation datasets: ScholarQABench-Multi contains only 108 questions spanning limited domains; Publication bias: Authors acknowledge 'publication bias, domain skew' in underlying literature may influence system behavior; Retrieval noise bias: Longer prompts with multiple retrieved passages negatively affect smaller models disproportionately

Open questions raised

Lack of standardized benchmark for question answering over scholarly knowledge graphs (SemOpenAlex)
Improved retrieval robustness needed, particularly under domain shift
Expanded domain coverage beyond current corpus limitations
Post-generation verification and citation validation mechanisms needed
More accurate routing strategies and lightweight reranking improvements required
Broader evaluations assessing usability, readability, and trust in real-world scholarly workflows

Data: ScholarQABench-Multi (Asai et al., 2024); PubMedQA (Jin et al., 2019); SciTLDR (Cachola et al., 2020); unarXive corpus (Saier et al., 2023) - 165K publications; SemOpenAlex knowledge graph (Färber et al., 2023); unarXive corpus (Saier et al., 2023) - 165K open-access publications; ScholarQABench-Multi (Asai et al., 2024) - 108 multi-document QA questions; PubMedQA (Jin et al., 2019) - biomedical question answering dataset; SciTLDR (Cachola et al., 2020) - scientific text compression benchmark; unarXive (Saier et al., 2023) - 165K open-access arXiv publications; SemOpenAlex (Färber et al., 2023) - scholarly knowledge graph with 26 billion RDF triplesCode: https://github.com/faerber-lab/lightweight-scholarly-qaExtracted from: pdfAgreement 56%

Explore related topics

RAG for Research Literature Discovery Experimental