AI evidence extraction

Anything you can do, A-I can do better... Or can it? Comparing ChatGPT's Search Strategy Outputs with Cochrane Review Searches

Rebecca Carlson, Katherine Howell, Elizabeth Moreton, Emily Jones · UNC Libraries · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.17615/3drp-qx72

Methodology & findings

Study design

Comparative measurement study comparing ChatGPT-generated search strategy outputs with Cochrane Review searches

Main result

The abstract indicates that the authors "measured to what extent ChatGPT could help develop comprehensive literature search strategies," investigating ChatGPT's capabilities for completing specific literature search tasks such as term generation, database syntax, and search hedge formatting, though the abstract notes that "generalizability is lacking" in previous studies.

Reports effect sizes.

Research paradigm

Empiricist/Positivist

Author conclusions

The abstract suggests the authors are investigating "improving efficiency with GenAI" in systematic review workflows, with the stated motivation that "designing literature searches for systematic reviews is time-consuming, even for experienced librarians, so improving efficiency with GenAI is a possibility worth investigating."

Risk of bias

Potential selection bias in choice of Cochrane reviews for comparison; Possible technology bias given rapid evolution of GenAI tools; No mention of blinding in methodology; Lack of blinded assessment mentioned in abstract

Limitations

The abstract notes that "generalizability is lacking" in previous studies that "measured ChatGPT's capabilities for completing specific literature search tasks, such as term generation, database syntax, and search hedge formatting," suggesting this study aims to address but may still have scope limitations.

Open questions raised

The authors identify that while "previous studies have measured ChatGPT's capabilities for completing specific literature search tasks," there is a gap in understanding how ChatGPT performs on comprehensive search strategy development and generalizability across different review contexts.
The authors identified that "Previous studies have measured ChatGPT's capabilities for completing specific literature search tasks, such as term generation, database syntax, and search hedge formatting, but generalizability is lacking," indicating a gap in comprehensive evaluation of ChatGPT for full search strategy development.
The authors identify that "previous studies have measured ChatGPT's capabilities for completing specific literature search tasks, such as term generation, database syntax, and search hedge formatting, but generalizability is lacking," indicating a gap in understanding ChatGPT's broader applicability to comprehensive literature search strategy development.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 68%

Explore related topics

Systematic Review Automation Literature Discovery Empirical Study