AI evidence extraction

Applying LLMs and Semantic Technologies for Data Extraction in Literature Reviews

Camille Demers · Proceedings of the Annual Conference of CAIS / Actes du congrès annuel de l ACSI · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.29173/cais2020

Methodology & findings

Study design

Pilot study using lightweight LLMs (Mistral Small 3.1 and GPT-4o mini) to extract data from open-access publications included in existing systematic literature reviews.

Main result

The study found that "The modest ROUGE scores observed for both models are partially due to the rigidity of this measure, which relies on strict word overlap between the models' predictions and the ground truth, rather than on semantic similarity." However, "cosine similarity measures suggest a strong semantic correspondence between model predictions and ground truth (0.8619 for Mistral Small 3.1 and 0.8508 for GPT-4o mini)." The findings varied across data elements, with shorter data elements like Country achieving higher performance (0.9422 for Mistral Small 3.1, 0.9628 for GPT-4o mini) than more complex elements like Motivational factors toward professional development.

Research paradigm

Pragmatist/mixed methods (combines computational evaluation with qualitative methodological framing)

Author conclusions

The authors conclude that "This pilot study explored the use of semantic and AI technologies to support ontology-based data extraction for literature reviews in the social sciences. Using lightweight LLMs yielded insightful yet preliminary results in extracting structured data from a sample of 41 studies in LIS." They further state that "Ultimately, this work highlights some of the challenges and opportunities these technologies offer to support knowledge synthesis."

Risk of bias

Selection bias: only 4 reviews selected from 30 screened articles; Language bias: only English-language reviews included; Publication bias: only open-access studies retained (41 of 70 studies); Domain bias: sample limited to Library and Information Science field; Prompt development bias: prompts developed on subset from single review; Selection bias: Only PRISMA-compliant reviews from Library and Information Science were included, limiting generalizability; Publication bias: Only open-access studies were included to avoid proprietary license issues, potentially skewing results; Sample size bias: Very small sample (4 reviews, 41 studies) for a pilot study; Domain-specific bias: Reviews selected exclusively from Library and Information Science field; Prompt development bias: Prompts were developed on one review and tested on three reviews, creating potential overfitting; Selection bias: Only Library and Information Science reviews included; excluded non-LIS fields; Publication bias: Only PRISMA-compliant reviews selected; Language bias: English-language publications only; Access bias: Only open-access studies included (41 of 70 available); Sample size limitation: Only 4 reference reviews selected, 6 studies for prompt development

Limitations

The authors state that "The limited scope of this work raises questions about the generalizability of these methods across diverse fields within social sciences." Additionally, "The evaluation was restricted to the data elements specific to the reference systematic reviews, as DoCO elements did not always clearly correspond to the review-specific data elements, making it difficult to establish ground truth for DoCO-based extraction." They also acknowledge that "Given the limitations of the evaluation methods used in this preliminary study, future work will aim to explore the use of manual or qualitative evaluations to better highlight the potential and the challenges of using LLMs for scientific information extraction."

Open questions raised

Generalizability of methods across diverse fields within social sciences
Refinement of proposed evaluation workflow
Exploration of visualization functionalities through triple store integration
Incorporation of additional language models
Manual or qualitative evaluations to better highlight potential and challenges of using LLMs for scientific information extraction
The authors identify the following future directions: (1) "refining the proposed evaluation workflow"; (2) "exploring visualization functionalities through triple store integration"; (3) "incorporating additional language models"; (4) exploring manual or qualitative evaluations to better assess potential and challenges of LLMs for scientific information extraction; (5) investigating generalizability across diverse fields within social sciences.

Data: LISTA database (Library, Information Science & Technology Abstracts) - accessed January 2025; LISTA (Library, Information Science & Technology Abstracts) database - source of systematic reviews; Unpaywall REST API - used for identifying open access studiesExtracted from: pdfAgreement 63%

Explore related topics

Systematic Review Automation Literature Discovery Case Study