AI evidence extraction

CG-RAG: Research Question Answering by Citation Graph Retrieval-Augmented LLMs

Yuntong Hu, Zhihan Lei, Z. G. Dai, Allen A. Zhang, Abhinav Angirekula, Zheng Zhang et al. · ArXiv.org · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2501.15067

Methodology & findings

Study design

Empirical evaluation study using two datasets (PubMedQA-1k with 7,849 papers in citation graph; PapersWithCodeQA with 12,171 papers and 924 questions).

Sample

N = 1924, 4 groups

Primary method

Performance metrics computed include Hit@1, Hit@3, Mean Reciprocal Rank (MRR), Accuracy (Acc), F1 score, and UniEval evaluation metrics (Coherence, Consistency, Relevance). Software: Graph Transformer as graph encoder, GPT-4 for generation (gpt-4o-2024-05-13 model), trained using AdamW optimizer on NVIDIA A10 GPUs.

Main result

The proposed Contextualized Graph Retrieval-Augmented Generation with LeSeGR achieves state-of-the-art performance across all tasks and datasets. Specifically, "LeSeGR achieves a Coherence score of 0.956 on PapersWithCodeQA, outperforming ColBERT's 0.927," and "our method achieves superior retrieval accuracy by effectively entangling sparse and dense signals within the graph structure, allowing contextual information to enhance relevance scoring." Additionally, "LeSeGR achieves the highest Hit@1 and Hit@3 scores" on the citation graph of PubMed with Hit@1 of 0.961 and Hit@3 of 0.987.

Reports effect sizes.

Research paradigm

Empirical - computational/algorithmic evaluation

Author conclusions

The authors conclude: "By leveraging a contextual citation graph, our approach effectively captures intra- and inter-document relationships, enabling a dynamic propagation of contextual information through an entangled hybrid retrieval paradigm." They further state that "Extensive experiments across multiple citation networks demonstrate the superior performance of CG-RAG based on LeSeGR, achieving state-of-the-art results in retrieval metrics such as Hit@1 and generation metrics such as Coherence and Relevance." They conclude that "our findings underscore the effectiveness of graph-contextualized representations in advancing the capabilities of retrieval-augmented generation for citation graphs, setting a new benchmark for retrieval-augmented research question answering."

Risk of bias

Dataset construction bias: PubMedQA-1k citation graph was constructed post-hoc by the authors rather than using pre-existing gold-standard citation data; Evaluation metric selection: Use of proprietary UniEval model for generative task evaluation may not be reproducible; Baseline selection: Comparison limited to specific baseline variants; other graph-based RAG approaches may exist; Hyperparameter tuning: Authors specify optimal settings (e.g., Top-4 context) but do not discuss sensitivity analysis across full parameter space; GPU-dependent results: Efficiency measurements conducted on specific hardware (NVIDIA A10 GPUs), potentially not generalizable

Limitations

The paper acknowledges that "the performance of LeSeGR appears to be primarily constrained by the quality of the dense retrieval signal." Additionally, while the authors constructed citation graphs for PubMedQA-1k by extracting references, the original PubMedQA dataset "lacks citation information between papers, which we addressed by extracting the references for each paper." The study is limited to two specific datasets and does not explicitly discuss limitations regarding generalization to other domains or citation graph structures.

Open questions raised

The authors do not explicitly state identified gaps, but their work implicitly addresses the gap that "most existing literature question answering methods fail to incorporate this structural context effectively" regarding citation graph structures in RAG systems.
The paper identifies the gap that while Retrieval-Augmented Generation (RAG) has been advanced with techniques like GraphRAG, "most existing literature question answering methods fail to incorporate this structural context effectively." The authors note that "designing a retrieval system that is both lexically and semantically aware in graph retrieval remains an unresolved challenge" and that traditional retrieval methods "fall short, as they fail to account for intricate interdocument relationships."
The paper addresses the gap that traditional retrieval methods fail to account for intricate inter-document relationships and citation links in paper citation networks. It identifies that designing retrieval systems that are both lexically and semantically aware in graph retrieval remains an unresolved challenge.

Data: PubMedQA-1k: 1,000 question-answer pairs with 7,849 papers in constructed citation graph; PapersWithCodeQA: 924 questions from 84 leaderboards with 12,171 papers from arXiv (420 True/False, 420 multiple-choice, 84 generative questions); PubMedQA-1k dataset (publicly available, introduced by Jin et al., accessed via PubMed: https://pubmed.ncbi.nlm.nih.gov/); PapersWithCodeQA dataset (collected from PapersWithCode website: https://paperswithcode.com/); PubMedQA-1k: publicly available dataset with 1,000 question-answer pairs and citation graph with 7,849 papers; PapersWithCodeQA: collected from Papers with Code website, 924 questions from 84 leaderboards with 12,171 papersExtracted from: pdfAgreement 62%

Explore related topics

RAG for Research Literature Discovery Benchmarking