AI evidence extraction

VAIV bio-discovery service using transformer model and retrieval augmented generation

Seonho Kim, Juntae Yoon · BMC Bioinformatics · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

1.27

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s12859-024-05903-6

Methodology & findings

Study design

Design science approach combining transformer-based NLP (named entity recognition and relation extraction), neural retrieval augmented generation (RAG), and large language models (LLM).

Primary method

Design science research with modular architecture incorporating: (1) named entity recognition and relation extraction modules, (2) indexing module for efficient retrieval, (3) search and answer generation module using RAG

Main result

The system demonstrated high QA performance with a ROUGE-1 score of 0.912 (F-score) and a BLEU score of 0.795 using only the 2023 PubMed baseline. From 226,241 abstracts, "85,006 diseases, 167,804 chemical compounds/drugs, 143,042 proteins/genes were recognized. Additionally, 663,732 (CPR), 151,193 (DDI), and 302,091 (CDR) pairs were ultimately identified from 2.12 million sentences as exhibiting specific interactions between the entities after excluding recognized false interactions."

Research paradigm

Design science; computational/empirical

Author conclusions

In this work, we introduced a novel biomedical search system that incorporates biomedical entity and relation extraction. It provides an efficient way to find biomedical entities and relations associated with a specific entity from scientific literature. In addition, the system gives an answer for a natural language query through neural search and summarization by RAG (retrieval augmented generation) and LLM. This enables researchers or curators to quickly grasp the research findings of interest within a large collection of research papers. We also emphasize the significance of biomedical text mining and sophisticated search techniques in discovering valuable information from the vast amount of unstructured text data generated in biomedical research.

Risk of bias

Selection bias in source documents: limited to PubMed abstracts (2023 baseline + daily updates), excluding full-text articles and other biomedical databases; Training data bias: T5 slim_dec model trained on ChemProt and DrugProt datasets which may not represent full spectrum of biomedical relations; Entity recognition bias: BioBERT tagger achieved F1-score of 0.86, indicating systematic false negatives/positives in entity extraction; Incomplete relation type coverage: CDR interactions limited to 'potential' only due to insufficient training datasets; CPR:7 and CPR:8 categories noted as having limited training data; Vocabulary/nomenclature bias: acknowledged challenge with entity synonyms, abbreviations, and diverse nomenclature causing inconsistencies; Language bias: system designed for English-language biomedical literature only; Selection bias: Limited to 2023 PubMed abstracts and TTD documents only; does not include full-text articles; NER tagger limitations: Entity recognition tool achieved F1-score of 0.86, introducing potential false negatives/positives; Training data limitations: CDR training dataset comprised only 1,500 abstracts, limiting generalizability for chemical-disease relations; Incomplete interaction detection: System cannot identify interactions mentioned across multiple documents or in full texts; Limited training data for certain interaction types (CPR:7 and CPR:8); Potential NER tagger errors requiring manual dictionary augmentation; Limited CDR dataset availability affecting CDR interaction extraction; Entity ambiguity in biomedical text (e.g., entities that are both drugs and proteins)

Limitations

The 2023 annual baseline of 219,317 and 6,924 daily update abstracts from PubMed might not provide comprehensive, high-quality information that we need
The number of recognized interactions is significantly smaller than in CTD because CTD includes inferred interactions, and we only consider abstracts, not the full texts of publications
In practice, even with a large collection of papers, some crucial knowledge may only be mentioned in a very few papers, making it difficult to discover
Thus, there are inherent limitations in relying solely on research papers to extract important knowledge
For CDR task, only potential interactions are recognized, due to the insufficient datasets for training more detailed and specific interaction types.

Open questions raised

Need for comprehensive, high-quality biomedical literature beyond PubMed abstracts; intention to expand to arXiv (2.4 million articles) and full-text PubMed articles via E-utilities API
Improvement of neural search through filters like citation counts, journal prestige, and impact factor
Expansion of named entity types: cell lines, cell types, species, biological processes, body parts
Expansion of interaction types: Gene-Gene Interactions, Protein-Protein Interactions, Gene-Disease Associations, Protein-Disease Associations, Drug-Metabolite Interactions
Improved training data availability: better construction and sufficiency of training datasets for relation extraction, especially for CDR interaction types
Further research on NE tagging and relation extraction accuracy

Data: PubMed 2023 baseline (219,317 abstracts) - freely available via NLM; PubMed daily updates (6,924 abstracts); Therapeutic Target Database (TTD) documents (8,499 targets); BioASQ Task B Task11B-GoldenEnriched dataset (330 questions, used 258); ChemProt corpus (training/evaluation dataset for relation extraction); DDI corpus (training/evaluation dataset); DrugProt corpus (training/evaluation dataset); PGxCorpus (evaluation dataset for NER); BioCreative CDR task dataset (1,500 PubMed abstracts); PubMed 2023 baseline and daily updates (https://www.ncbi.nlm.nih.gov/pubmed); Therapeutic Target Database (TTD) - 8,499 targets with associated documents; BioASQ Task B datasets (258 questions from Task11B-GoldenEnriched dataset, 330 questions total); ChemProt BioCreative dataset (training data for relation extraction); DrugProt Corpus (training data for relation extraction); DDI dataset (training data for drug-drug interaction extraction); Chemical-Disease Relation (CDR) task dataset (1,500 PubMed abstracts); 2023 PubMed baseline (219,317 abstracts); TTD (Therapeutic Target Database) documents (8,499 targets); BioASQ Task B datasets (258 questions from Task11B-GoldenEnriched dataset); ChemProt corpus; DDI corpus; DrugProt corpus; PGxCorpusCode: https://github.com/libra_iry/bio-ner (BioBERT-based NER tagger reference); https://github.com/libra/bio-ner (biomedical NER tool reference); https://github.com/libra iry/bio-nerExtracted from: pdfAgreement 56%

Explore related topics

RAG for Research Literature Discovery Design Science