AI evidence extraction

LitFM: A Retrieval Augmented Structure-aware Foundation Model For Citation Graphs

Jiasheng Zhang, Ali Maatouk, Jialin Chen, Ngoc Bui, Qianqian Xie, Leandros Tassiulas et al. · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

3.52

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1145/3711896.3737028

Methodology & findings

Study design

Mixed methods: (1) Graph neural network design with self-supervised contrastive learning for the retriever component; (2) Multi-task instruction fine-tuning using LoRA on curated citation graphs from arXiv and PubMed; (3) Quantitative evaluation on six literature tasks (citation link prediction, paper recommendation, title generation, abstract completion, citation sentence generation, related work generation) across three domain-specific datasets; (4) Human evaluation study with active researchers (n=multiple, not specified) rating generated related work sections; (5) Ablation studies and comparative analysis against multiple baselines..

Primary method

Design science research combining artifact engineering with empirical evaluation through benchmarking and human studies. Iterative development of components (graph retriever, instruction tuning paradigm, inference pipeline).

Main result

LitFM demonstrates state-of-the-art performance across literature-related tasks. As the authors state, "LitFM outperforms leading models (e.g., GPT-4o) across literature-related tasks, such as citation link prediction and paper recommendation." On the medicine dataset specifically, "LitFM achieves 8.0% improvement in accuracy for citation link prediction and 5.9% in Hits@1 for citation recommendation." The model also shows superior performance on related work generation, where "LitFM outperforms existing powerful LLMs on related work generation in both BERT and ROUGE scores," with improvements of "1.6% on the CS dataset and 9.8% on the Physics dataset" compared to GPT-4o.

Research paradigm

Design science / empirical validation

Author conclusions

The authors conclude: "In this paper, we develop the first literature foundation model LitFM that uniformly handles various citation graph-related tasks. It consists of a graph retriever and a knowledge-infused LLM to avoid hallucinations and generalize to handle new papers outside the citation graph. Experimental results demonstrate both the versatility of LitFM and its superior performance across six benchmark tasks." They also state that their work addresses the critical need for "a unified model capable of effectively performing diverse literature tasks-moving beyond a narrow focus on singular tasks-while mitigating hallucination issues, delivering high performance, and addressing the Matthew effect."

Risk of bias

Matthew effect inherent in citation graphs (bias toward highly-cited papers); Degree bias in message-passing mechanism favoring high-degree nodes; Dataset composition bias: CS (340k nodes), Physics (59k nodes), Medicine (2.1M nodes) - imbalanced domain representation; Potential selection bias in human evaluation study (participants were 'active researchers in the AI domain'); Training/test set split bias: connected dense subgraph sampling may not represent broader citation patterns; Citation graph bias: Matthew effect (highly cited papers disproportionately recommended); Degree bias in graph neural networks favoring high-degree nodes; Dataset selection bias: citation graphs from arXiv and PubMed may not represent all domains; Evaluation bias: human evaluators from AI domain may not represent all research fields; Selection bias in dataset construction: Citation graphs from arXiv and PubMed may not represent all research domains equally; Degree bias in the retriever: The message-passing mechanism may prioritize highly cited papers; Matthew effect amplification: Initial tendency to recommend well-established papers over emerging research; Domain-specific sampling: Test set of 2,000 connected dense subgraphs may not be representative of overall graph distribution; Human evaluation bias: Active researchers in AI domain may have different preferences than researchers in other fields; Baseline selection: Some baselines (e.g., GAT with BERT embeddings) may not be optimally tuned

Limitations

The authors acknowledge: "However, one area where LitFM lags behind is English structure and coherence, with a score of 4.2, compared with 4.6 for DeepSeek-R1 and 5 for GPT-4o
This is due to the significantly larger number of parameters in GPT-4o and DeepSeek-R1, leading to advanced proficiency in English compared with LitFM." Additionally, the authors note that "current state-of-the-art (SOTA) graph-based retrievals face two key limitations: 1) Their performance highly depends on the precision and informativeness of given queries..
2) They rely solely on semantic similarity for retrieving relevant papers, which exacerbates the Matthew effect."

Open questions raised

The paper identifies gaps in existing approaches: (1) current graph-based retrievals lack robustness to incomplete or ambiguous queries, (2) they rely solely on semantic similarity without diversity consideration, exacerbating the Matthew effect, (3) single-task approaches limit knowledge transfer across diverse literature tasks, and (4) lack of citation context in existing datasets limits adaptability.
The paper identifies the need for: (1) models that handle diverse literature-related tasks beyond single-task focus; (2) retrieval methods that balance accuracy with diversity to mitigate the Matthew effect; (3) improved handling of incomplete or ambiguous queries in citation retrieval; (4) unified approaches that internalize citation graph structure while maintaining knowledge transfer across tasks.
The paper identifies gaps in existing approaches: (1) Most domain-specific models focus on single tasks, limiting cross-task knowledge transfer; (2) Current RAG methods do not adequately address domain-specific citation graph properties; (3) Existing graph retrievers lack diversity in recommendations and suffer from the Matthew effect; (4) Previous citation-related datasets lack citation context and local information about how papers cite each other; (5) Related work generation methods lack controllability and either generate in a single step or rely on retrieved papers without personalized refinement.

Data: Three curated citation graph benchmarks from Medicine (PubMed Central), Computer Science (arXiv), and Physics (arXiv) - specific download links not provided in paper; Medicine dataset (PubMed Central repository): 2.1M nodes, 7.4M edges, 1.5M related works; Computer Science dataset (arXiv LaTeX sources): 340k nodes, 3.2M edges, 188k related works; Physics dataset (arXiv LaTeX sources): 59k nodes, 120k edges, 19k related works; Medicine dataset: Constructed from PubMed Central repository (2.1M nodes, 7.4M edges, 1.5M related works); Computer Science dataset: Constructed from arXiv LaTeX sources (340k nodes, 3.2M edges, 188k related works); Physics dataset: Constructed from arXiv LaTeX sources (59k nodes, 120k edges, 19k related works); Authors state: "We open-source LitFM, along with a graphical user interface, facilitating seamless model usage and enabling broader adoption in research and academic applications."Code: LitFM is open-sourced with a graphical user interface (specific GitHub/repository URL not provided in paper); LitFM is open-sourced along with a graphical user interface (specific GitHub URL not provided in paper); The authors indicate code will be open-sourced but no specific repository URL is provided in the paper. Statement: "We open-source LitFM, along with a graphical user interface, facilitating seamless model usage and enabling broader adoption in research and academic applications."Extracted from: pdfAgreement 44%

Explore related topics

Systematic Review Automation Literature Discovery Design Science