AI evidence extraction

Multi-Stage LLM Pipeline to Support Qualitative Content Analysis – A Proof of Concept Experiment with Expert Validation

Eva Forster, Nadja Kartschmit, Elisabeth Klager, E. Mosor, Benjamin Schuster, Erika Mosor et al. · Studies in health technology and informatics · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.3233/shti260065

Methodology & findings

Study design

Proof-of-concept experimental design with expert validation.

Sample

N = 28, 5 groups

Main result

The pipeline produced "12 higher-level and 73 lower-level concepts in 45 minutes, demonstrating substantial efficiency gains compared to manual analysis." Expert assessment confirmed "high content validity, strong thematic overlap with manual results, and all outputs traceable to source text," with "the majority of evaluators" deeming "outputs suitable for scientific use following minor revisions."

Reports effect sizes.

Research paradigm

Pragmatist/Mixed-methods (combining computational automation with qualitative expert validation)

Author conclusions

"LLM-assisted qualitative analysis, embedded in a transparent pipeline and subject to expert oversight, interpretation and contextualisation, can produce verifiable, high-quality results and substantially enhance the scalability of qualitative research."

Risk of bias

Small evaluator sample (five researchers); Potential evaluator bias as they conducted original manual analysis; Single domain focus (health data donation interviews); Limited generalizability to other qualitative research contexts; Potential evaluator bias (five researchers evaluated outputs from analysis they originally conducted manually); Selection bias in transcript sampling (28 interviews on single topic: health data donation); Limited transparency on QUEST framework criteria applied; No specification of inter-rater reliability metrics among the five evaluators; Selection bias: The five expert evaluators may be biased toward the pipeline if they were involved in its design or have vested interest in its success; Limited generalizability: Single study domain (health data donation) and relatively small transcript sample (n=28); Evaluator expertise bias: Results depend on the five researchers' assessment criteria and their familiarity with the pipeline; No independent blinded evaluation reported

Open questions raised

Concerns about transparency, reproducibility, and methodological validity of LLMs in scientific research have limited their adoption; the paper addresses this gap by demonstrating an auditable and expert-validated approach.
The authors identify the need for LLM solutions that address "concerns about transparency, reproducibility, and methodological validity" to enable broader scientific adoption of computational approaches in qualitative research.
The abstract identifies the gap that "manual analysis is time-intensive and difficult to scale, particularly in larger datasets" and that "concerns about transparency, reproducibility, and methodological validity have limited" LLM adoption in science. Future work would likely address broader validation across diverse qualitative research domains.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 61%

Explore related topics

Qualitative Coding with AI Case Study