AI evidence extraction

Multi-Stage LLM Pipeline to Support Qualitative Content Analysis – A Proof of Concept Experiment with Expert Validation

Eva Forster, Nadja Kartschmit, Elisabeth Klager, E. Mosor, Benjamin Schuster, Erika Mosor et al. · Studies in health technology and informatics · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.3233/shti260065

Methodology & findings

Study design

Proof-of-concept experimental design with expert validation.

Sample

N = 28, 2 groups

Main result

The pipeline produced "12 higher-level and 73 lower-level concepts in 45 minutes, demonstrating substantial efficiency gains compared to manual analysis." Expert assessment confirmed "high content validity, strong thematic overlap with manual results, and all outputs traceable to source text," with "the majority of evaluators" deeming "outputs suitable for scientific use following minor revisions."

Reports effect sizes.

Research paradigm

Pragmatist/Mixed-methods (combining computational automation with qualitative expert validation)

Author conclusions

"LLM-assisted qualitative analysis, embedded in a transparent pipeline and subject to expert oversight, interpretation and contextualisation, can produce verifiable, high-quality results and substantially enhance the scalability of qualitative research."

Risk of bias

Small evaluator sample (five researchers); Potential evaluator bias as they conducted original manual analysis; Single domain focus (health data donation interviews); Limited generalizability to other qualitative research contexts; Limited transparency on QUEST framework criteria applied; No specification of inter-rater reliability metrics among the five evaluators; Selection bias: The five expert evaluators may be biased toward the pipeline if they were involved in its design or have vested interest in its success; No independent blinded evaluation reported

Open questions raised

Concerns about transparency, reproducibility, and methodological validity of LLMs in scientific research have limited their adoption
the paper addresses this gap by demonstrating an auditable and expert-validated approach.

Data: not_statedCode: not_statedExtracted from: pdf

Explore related topics

Qualitative Coding with AI Case Study