AI evidence extraction

Trend Analysis and LLM Agent-Generated Report of ChatGPT Applications in Healthcare

Po Ju Chou, Yu Lin, Eldon Y. Li · Journal of the Association for Information Systems · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Proof-of-concept (POC) study using LLM agents with agentic Retrieval-Augmented Generation (RAG) technology to conduct meta-analysis of medical journal articles and identify trends in ChatGPT applications.

Main result

The findings reveal that "LLMs can efficiently extract and synthesize basic and contextual information from numerous documents, addressing the growing challenge of timely evidence synthesis in medicine." Additionally, "Trend analysis via LLMs enables broader and deeper insights than traditional human reviews, facilitating knowledge discovery, minimizing human bias, and enhancing resource allocation and policy decisions."

Research paradigm

pragmatic/applied AI methodology

Author conclusions

The study "underscores the developmental potential of LLM tools for medical literature review and their role in advancing healthcare research, education, and practice." The authors demonstrate that LLMs and agentic RAG technology can effectively support evidence synthesis in medicine while acknowledging that challenges in interpretability and clinical relevance remain to be addressed.

Risk of bias

Potential bias from LLM training data limitations; Clinical relevance concerns in automated synthesis; Interpretability challenges in machine-generated summaries; Human bias minimization claims require validation; Interpretability limitations of LLMs; Potential data limitations in document selection; Clinical relevance validation gaps; Potential bias from LLM hallucination or inaccuracy in information extraction; selection bias in journal article sampling; lack of human validation of LLM-generated syntheses; limited transparency in how the LLM agents processed and prioritized information.

Limitations

While the study demonstrates potential, "challenges such as interpretability, data limitations, and clinical relevance remain" as acknowledged limitations in the application of LLM tools for medical literature review.

Open questions raised

The authors identify the need to address interpretability, data limitations, and clinical relevance challenges in LLM-based medical literature review systems.
Future work should address interpretability of LLM outputs, validate clinical relevance of LLM-generated insights, establish data quality standards for LLM-based literature analysis, and develop frameworks for integrating LLM agents into clinical decision-making workflows.
Future work should address interpretability of LLM decision-making, improve data quality and scope, strengthen clinical relevance assessment, and develop validation frameworks for LLM-generated medical insights.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 62%

Explore related topics

Meta-Analysis Automation Knowledge Synthesis Empirical Study