AI evidence extraction

What Should I Cite? A RAG Benchmark for Academic Citation Prediction

Leqi Zheng, Jiajun Zhang, Canzhi Chen, Chaokun Wang, Hongwei Li, Yuying Li et al. · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

4.85

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1145/3774904.3792075

Methodology & findings

Study design

Computational benchmark study involving: (1) construction of a three-level hierarchical corpus of 554,719 papers from Google Scholar; (2) curation of 7,267 samples for Task 1 (list-specific citation prediction) and 8,541 samples for Task 2 (position-specific citation prediction); (3) implementation of a multi-level hybrid retrieval-augmented generation (RAG) pipeline with fine-tuned embedding models using contrastive learning; (4) comprehensive evaluation across multiple baseline methods including closed-source LLMs (GPT-o3, GPT-5, Claude-3.5, Gemini-2.5-Pro, Grok-3) and open-source models (DeepSeek-V3, Qwen3); (5) four experimental settings: few-shot in-context learning, RAG, supervised fine-tuning, and supervised fine-tuning with RAG..

Main result

The study demonstrates that "RAG consistently improves performance across all models, with retrieval depth significantly impacting prediction quality" and that "CitationGenerator-30B (R=10) achieving the highest performance: Recall@20 of 0.076, NDCG@20 of 0.367 for Task 1, and PACA@20 of 0.303 for Task 2." The results further show that "supervised fine-tuning demonstrates remarkable effectiveness, as CitationGenerator-30B without RAG achieves competitive performance with leading closed-source models that use retrieval."

Research paradigm

empirical-computational

Author conclusions

The authors conclude: "We propose CiteRAG, the first comprehensive RAG-integrated benchmark for evaluating large language models on academic citation prediction. The benchmark features dual-granularity tasks, a hierarchical corpus of 554k papers, and standardized evaluation framework. Extensive experiments across state-of-the-art language models demonstrate that RAG integration consistently enhances prediction accuracy and reduces hallucination rates. Our results reveal that specialized domain adaptation through supervised fine-tuning provides substantial performance gains, while retrieval augmentation offers complementary benefits through external knowledge grounding. Our open-source toolkit provides the research community with reproducible evaluation protocols and comprehensive baselines, establishing a methodological foundation for advancing citation prediction systems."

Risk of bias

Dataset collection bias: papers sourced only from Google Scholar, which may not represent all academic domains equally; Temporal bias: collection over past decade may underrepresent emerging research areas; Language bias: likely English-centric collection from Google Scholar; Model selection bias: choice of specific embedding models and LLMs may affect generalizability; Citation pattern bias: papers excluded with non-standard or excessive citation patterns (>10 of same reference per section); Selection bias in corpus collection from Google Scholar (may not represent all academic domains equally); potential bias toward papers with standard citation formats; data contamination risk mitigated by removing processed papers from corpus; noise robustness analysis indicates vulnerability to retrieval errors above 40% noise levels.; Selection bias: Papers collected from Google Scholar may not represent all academic domains equally; Data contamination risk: Papers used in test sets were permanently removed from corpus to prevent leakage; Temporal bias: Data collection spanned past decade but may not equally represent recent publications; Domain bias: Multidisciplinary collection across 10 fields may have unequal representation; Retrieval noise: Evaluation includes noise robustness testing but real-world retrieval performance variability; Model bias: Closed-source models have different training data and potential biases not disclosed

Limitations

The authors identify several limitations: "Figure 8 illustrates two challenging scenarios that expose current system limitations
The left panel shows a complex nested clause case where the citation appears within a deeply embedded subordinate clause, causing models to focus on the main clause about generalization rather than the nested context discussing momentum-based updates, resulting in incorrect predictions
The right panel demonstrates technical detail overload, where multiple scattered concepts in a lengthy sentence distract models from identifying the correct transformer mechanism reference." Additionally, the task formulation itself has constraints, as the approach "employs a fixed-length prefix chunking strategy, using title and abstract (typically 200-400 tokens) to represent each paper," which addresses practical constraints but may limit comprehensive evaluation.

Open questions raised

Absence of comprehensive benchmarks for citation prediction evaluation
Oversimplified task formulations that diverge from practical user scenarios
Limited frameworks for evaluating hierarchical structure of scientific literature
Lack of systematic evaluation of advanced approaches such as RAG
Need for evaluation of hallucination and diversity in citation prediction systems
Handling of complex syntactic structures in citation contexts (nested clauses, technical detail overload)

Data: CiteRAG benchmark dataset with 554,719 papers, available at https://github.com/LQgdwind/CiteRAG; Task 1 evaluation dataset: 7,267 samples for list-specific citation prediction; Task 2 evaluation dataset: 8,541 samples for position-specific citation prediction; CiteRAG benchmark dataset (554,719 papers with 7,267 Task 1 and 8,541 Task 2 evaluation samples) - https://github.com/LQgdwind/CiteRAG; CiteRAG benchmark dataset (554,719 papers, 7,267 Task 1 samples, 8,541 Task 2 samples) - available at https://github.com/LQgdwind/CiteRAGCode: https://github.com/LQgdwind/CiteRAG - source code and data for the benchmark; https://github.com/LQgdwind/CiteRAG; https://github.com/LQgdwind/CiteRAG - source code and data for CiteRAG benchmarkExtracted from: pdfAgreement 50%

Explore related topics

RAG for Research Literature Discovery Benchmarking