AI evidence extraction

A Retrieval-Augmented Generation Framework for Academic Literature Navigation in Data Science

Ahmet Yasin Aytar, Kemal Kılıç, Kamer Kaya · arXiv (Cornell University) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2412.15404

Methodology & findings

Study design

Design science research with experimental evaluation using the RAGAS (Retrieval-Augmented Generation Assessment System) framework.

Primary method

Design science research with systematic component evaluation and ablation testing

Main result

The study found that "the proposed Enhanced RAG Architecture, which integrates fine-tuning on domain-specific datasets, employs GROBID for data structuring, implements Semantic Chunking, and utilizes an Abstract-First strategy, significantly enhances the capability of the RAG pipeline to retrieve pertinent academic content." Additionally, "Models that were fine-tuned using domain-specific textbooks, particularly in Experiments 2 and 5, consistently outperformed the baseline model in retrieving context relevant to the query," and the "Abstract-First strategy introduced in the pipeline further refined the retrieval process by focusing on abstracts to filter relevant content before conducting a more in-depth full-text search. This approach led to notable improvements in CR, achieving the highest scores across all configurations."

Research paradigm

Pragmatist (design science applied to information retrieval)

Author conclusions

"The proposed Enhanced RAG Architecture, which integrates fine-tuning on domain-specific datasets, employs GROBID for data structuring, implements Semantic Chunking, and utilizes an Abstract-First strategy, significantly enhances the capability of the RAG pipeline to retrieve pertinent academic content. The fine-tuning models with larger and more diverse training sets, as demonstrated in Experiments 3 and 6, were particularly effective in increasing Context Relevance (CR), which aligns with the study's goal of improving retrieval of academic literature for data scientists. While Semantic Chunking and Abstract-First approaches further refined retrieval precision, the observed trade-offs in Faithfulness suggest a need for future work to optimize how retrieved content is utilized in generating responses."

Risk of bias

Selection bias: Custom test set of 50 questions may not be representative of all data science use cases; Evaluator bias: RAGAS framework relies on LLM-based evaluation which introduces subjectivity; Dataset bias: Fine-tuning datasets drawn from 5-17 textbooks may not capture emerging research; Confounding: Comparison of multiple factors (GROBID, fine-tuning, semantic chunking, abstract-first, prompting) without fully controlled ablation; Generalizability concern: Questions designed for financial, agriculture, healthcare domains; unclear if applicable to other fields; Subjectivity in LLM-based evaluation (RAGAS framework uses LLMs as evaluators); Custom test set bias—questions designed specifically for this study may not generalize; Lack of standardized ground truth for evaluation; Selection bias in article retrieval from arXiv (only ~100 articles per question); Evaluation relies on LLM-based RAGAS framework which introduces subjectivity; Custom test set not validated against external benchmarks; Limited generalizability due to domain-specific question design; Potential selection bias in article retrieval from arXiv API

Limitations

"One key challenge is the lack of a standardized ground truth for evaluating the quality of the generated answers
To address this, we utilized the RAGAS framework, which relies on LLMs for evaluation
While this method was the most viable given the circumstances, it introduces a degree of subjectivity into the assessment, potentially affecting the interpretation of the results
Additionally, the reliance on a custom test set, developed specifically for this study, poses another limitation
Although the questions were designed to reflect realistic challenges in data science, the lack of a widely accepted benchmark for RAG evaluation in academic literature restricts the generalizability of the findings."

Open questions raised

Need for standardized benchmarks and ground truth datasets for RAG evaluation in academic literature
Validation of abstract-first strategy effectiveness in domains where abstracts may not fully capture article nuances
Optimization of trade-offs between semantic chunking benefits and Faithfulness metric decline
Further improvements in prompt alignment and context-query combination
Application of knowledge graph-based approaches for automating validation and structuring of retrieved content
Further investigation needed to assess broader applicability of the abstract-first strategy across domains where abstracts may not fully capture article nuances

Data: arXiv API used to retrieve approximately 5,000 articles (~100 per question) (no explicit data release mentioned); Approximately 5,000 articles retrieved from arXiv API for constructing the vector database. The complete list of 50 test questions is provided in Appendix A.; Approximately 5,000 articles retrieved from arXiv API (100 articles per test question); custom test set of 50 questions; fine-tuning datasets from textbooks listed in Appendices B and CExtracted from: pdfAgreement 65%

Explore related topics

RAG for Research Literature Discovery Design Science