AI evidence extraction

MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature

Md Toyaha Rahman Ratul, Zhiqian Chen, Kaiqun Fu, Taoran Ji, Lei Zhang · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Large-scale benchmark construction using: (1) data collection from 15 official conference venues via venue-specific crawlers; (2) citation extraction using a two-tier pipeline combining Nougat (transformer-based PDF-to-markup) and GROBID (fallback); (3) three-tier annotation scheme with LLM-based judge (Gemini 2.5 Flash) validated against human expert labels on stratified sample (n=510); (4) baseline evaluation using sparse retrieval (BM25), dense scientific embeddings (SPECTER, SPECTER2, SciNCL, SciBERT, ColBERT), and graph-based methods (KTR, AR-GNN, LitFM); (5) primary evaluation metric: Recall@K with supplementary metrics MAP, MRR, nDCG@K, HR@K..

Main result

The study introduces MasterSet and establishes that "must-cite retrieval remains a challenging open problem." The authors found that "No method exceeds 50% Recall@100 on any ground truth type: even the strongest baseline, SciBERT-NTX, leaves more than half of must-cite papers unrecovered across all three tiers," with SciBERT-NTX achieving Recall@100 of 0.383, 0.410, and 0.493 on Types I, II, and III respectively, confirming that "must-cite retrieval is a strictly harder task than general citation recommendation."

Research paradigm

Empirical/Data-driven computational science

Author conclusions

The authors conclude that "must-cite retrieval remains a challenging open problem" with "substantial headroom for future methods," and that "must-cite recommendation is a strictly harder task than general citation recommendation: the relevant set is small, the recall requirement is high, and missing even one is consequential." They emphasize that MasterSet is "the first large-scale benchmark specifically designed to evaluate must-cite recommendation in the AI and ML literature" and that their three-tier labeling scheme provides "a principled, multi-faceted operationalisation of must-cite status."

Risk of bias

Closed-world assumption: genuinely overlooked citations cannot be labeled; LLM judge bias: position bias and systematic errors in LLM-generated labels, partially mitigated through prompt design; Venue selection bias: collection skewed toward highest-volume venues (NeurIPS, CVPR, AAAI, ICLR, ICML account for ~63% of papers); Temporal bias: dataset begins with deep learning era (2012-2013 onward); Domain bias: restricted to AI/ML literature only; Annotation bias: moderate inter-annotator agreement (Krippendorff's α = 0.461-0.492) indicates subjective judgment; Cold-start bias: 2025 test queries have no outbound citations affecting graph-based method evaluation; Closed-world assumption: genuinely overlooked citations cannot be labeled as positives; Selection bias in venue representation: skewed distribution toward high-volume venues (NeurIPS, CVPR, AAAI account for 35% of collection); LLM judge bias: position bias and parse error rates acknowledged from prior work; partially mitigated through structured single-digit output; Moderate inter-annotator agreement (α=0.461-0.492) suggests subjective disagreement on citation classification; Temporal bias: evaluation set limited to 2025 papers; training/pool from 2018-2024 may not reflect future research patterns; Coverage gaps: ECCV, ICCV biennial (alternate years); NAACL not held every year; earlier proceedings incomplete for some venues; LLM annotation bias: Gemini 2.5 Flash demonstrates systematic biases that could affect label quality despite human validation study; Selection bias: Dataset constructed from official proceedings only, excludes preprints and non-conference venues; Annotation bias: LLM-generated labels used as ground truth with inter-annotator agreement on human validation showing moderate agreement (α=0.461-0.492); Venue bias: Distribution skewed toward largest venues (NeurIPS, CVPR, AAAI, ICLR, ICML account for 63% of collection); Temporal bias: Majority of venues collected from 2012/2013 onward; earlier proceedings not comprehensively covered; Domain bias: Restricted to AI/ML literature; does not represent other scientific domains

Limitations

The paper states that "like all citation benchmarks, our framework shares the unavoidable closed-world property that genuinely overlooked citations cannot appear as positive labels," and notes that "graph-based methods underperform text-only baselines in the cold-start setting" because "2025 query papers have no outbound citations in the training pool, so the structural neighbourhood these models exploit is unavailable at test time—a property of the must-cite auditing scenario rather than a general indictment of graph-based retrieval." Additionally, inter-annotator agreement falls in the "moderate range" with Krippendorff's α of 0.461 for Type I and 0.492 for Type II.

Open questions raised

The paper identifies must-cite recommendation as an understudied problem. Authors note that "existing systems primarily focus on broad relevance rather than identifying the critical set of must-cite papers" and that no existing tool tells researchers which specific papers they must cite. The analysis reveals that graph-based methods underperform in cold-start settings and that citation-informed pre-training provides limited direct signal without task-specific supervision, indicating substantial headroom for future methods research.
Citation recommendation systems have primarily focused on broad relevance rather than identifying critical 'must-cite' papers; existing systems poorly matched to practical need of researchers to identify papers central to their contribution
No existing tool tells researchers which specific papers they must cite given a paper or idea
Distinction between sparse-label ground truths (Types I and III) relying on surface vocabulary versus semantic proximity (Type II) remains underexplored
Graph-based retrieval methods require access to citation graph structure, limiting applicability in cold-start scenarios where query papers have no outbound citations
Citation recommendation systems have predominantly focused on broad relevance rather than identifying critical 'must-cite' papers

Data: MasterSet: Available at https://anonymous.4open.science/r/masterset; MasterSet-CoreML-v1: Versioned benchmark split containing 153,373 papers from 15 venues, with 67,761 core-set papers (2018-2024) as candidate pool and 7,028 papers (2025) as evaluation set; MasterSet full collection: 153,373 papers across 15 venues; available at https://anonymous.4open.science/r/masterset; MasterSet-CoreML-v1: versioned benchmark split with 67,761 papers (2018-2024 pool) and 7,028 evaluation queries (2025); MasterSet: 153,373 papers from 15 venues; data and code available at https://anonymous.4open.science/r/masterset (as stated in abstract); MasterSet-CoreML-v1: Versioned benchmark split with 67,761-paper candidate pool (2018-2024) and 7,028 evaluation queries (2025)Code: https://anonymous.4open.science/r/masterset; Code and data available at: https://anonymous.4open.science/r/mastersetExtracted from: pdfAgreement 40%

Explore related topics

Systematic Review Automation Literature Discovery Benchmarking