AI evidence extraction

KnowRAG: A Zero-Shot Diagnostic Analysis of Knowledge Base Coverage in Scientific Retrieval-Augmented Generation

Assmaa Moutaoukkil, Ali El Mezouary, Kaoutar Boumalek · International Journal of Advanced Computer Science and Applications · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.14569/ijacsa.2026.0170422

Methodology & findings

Study design

Controlled experimental methodology with zero-shot comparative evaluation.

Main result

KnowRAG increased the average factual correctness of the baseline by more than double, from 27.5 percent to 59.5 percent. This improvement was consistent across all four test sets with gains ranging from 21 to 40 percentage points. "The Knowledge Gap Failure is the dominant failure mode, accounting for nearly half of all errors on average and a vast 82% of errors on QATestSet2." Additionally, "Generation Failure is extremely rare, accounting for only 4% of errors on average," suggesting that when correct context is provided, GPT-3.5-turbo performs faithfully without fine-tuning.

Research paradigm

Positivist/empiricist (quantitative evaluation of computational pipeline performance)

Author conclusions

The authors conclude: "This study presents KnowRAG, a zero-shot retrieval-augmented generation (RAG) pipeline that achieves notably better factual accuracy compared to a Large Language Model alone. Our analysis reveals that the main limitation in performance is not retrieval or generator quality, but rather the extent of Knowledge Base coverage, with missing knowledge responsible for most errors. These results shift the focus away from scaling models, suggesting that reliable scientific AI relies more on advanced data engineering than larger models." They further emphasize that "the effectiveness of an RAG system hinges on the quality of its knowledge base."

Risk of bias

Methodological circularity: LLM-as-a-Judge evaluation (GPT-4) assessing LLM generator (GPT-3.5) without human calibration; Limited corpus size: Knowledge base constructed from only 10 arXiv papers, raising generalizability concerns; Selection bias: Test sets derived from same corpus used for knowledge base construction; Evaluation bias: Single evaluator (GPT-4) without inter-rater reliability assessment; Chunking strategy bias: Fixed parameters (1,000-token chunks with 30-token overlap) may not be optimal for all document types; Circularity risk: Using GPT-4 to evaluate GPT-3.5 outputs without human expert calibration; Selection bias: Only ten arXiv publications used; limited to specific RAG-related papers; Architectural bias: Single pipeline configuration may not generalize to other RAG implementations; Temporal bias: Knowledge base built from recent RAG-related publications; limited coverage of other scientific domains; Methodological bias: Blind chunking strategy (RecursiveCharacterTextSplitter) may systematically disadvantage certain document types; Evaluation circularity: GPT-4 used to judge GPT-3.5-powered system (acknowledged by authors); Limited corpus size: Only ten arXiv papers used for knowledge base; Single architecture: Only one RAG pipeline architecture tested; Potential parametric contamination from GPT-3.5-turbo internal knowledge despite zero-shot prompting; Sampling bias in test set generation: All 240 questions generated by GPT-4 from same corpus; Format bias: Knowledge base chunking strategy may systematically fail on tabular data (as demonstrated in QATestSet2)

Limitations

The authors state: "Using an LLM (GPT-4) to judge a pipeline powered by an LLM (GPT-3.5) introduces potential circularity
While this study provides a diagnostic framework, these assessments have not yet been calibrated against human expert annotation." Additionally, "Our findings are derived from a single pipeline architecture and ten targeted articles
It remains unclear whether the 'coverage-over-generation' failure mode persists across different chunking strategies, larger corpora, or specialized scientific knowledge graphs."

Open questions raised

Need for structure-aware chunking strategies that preserve rhetorical organization of academic writing
Development of hybrid retrieval approaches combining dense and sparse methods to capture functional relevance beyond semantic similarity
Multimodal approaches specifically addressing tabular data and equations
Generalizability testing across different chunking strategies, larger corpora, and specialized scientific knowledge graphs
Calibration of LLM-as-a-Judge assessments against human expert annotation
Generalizability across different chunking strategies and larger corpora

Data: The complete knowledge base and generated test sets are stated as "available in the Appendix section to ensure full reproducibility" and "publicly available at link" (specific URL not provided in text). Four Q&A Test Sets (QATestSet1-4) each containing 60 question-answer-context triplets (total 240 triplets).; "The complete knowledge base and the generated test sets are available in the Appendix section to ensure full reproducibility." "To facilitate reproducibility, the four Q&A Test Sets files, including ground truth annotations, are publicly available at link." (specific URL not provided in text; noted as "link"); The authors state: "To facilitate reproducibility, the four Q&A Test Sets files, including ground truth annotations, are publicly available at link." However, the specific URL is not provided in the text. The complete knowledge base and generated test sets are stated to be "available in the Appendix section."Extracted from: pdfAgreement 56%

Explore related topics

RAG for Research Knowledge Synthesis Benchmarking