AI evidence extraction

LitFM: A Retrieval Augmented Structure-aware Foundation Model For Citation Graphs

Jiasheng Zhang, Ali Maatouk, Jialin Chen, Ngoc Bui, Qianqian Xie, Leandros Tassiulas et al. · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

Evidence

Citations

3.52

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1145/3711896.3737028

Methodology & findings

Study design

Mixed methods: (1) Graph neural network design with self-supervised contrastive learning for the retriever component; (2) Multi-task instruction fine-tuning using LoRA on curated citation graphs from arXiv and PubMed; (3) Quantitative evaluation on six literature tasks (citation link prediction, paper recommendation, title generation, abstract completion, citation sentence generation, related work generation) across three domain-specific datasets; (4) Human evaluation study with active researchers (n=multiple, not specified) rating generated related work sections; (5) Ablation studies and comparative analysis against multiple baselines..

Primary method

Design science research combining artifact engineering with empirical evaluation through benchmarking and human studies. Iterative development of components (graph retriever, instruction tuning paradigm, inference pipeline).

Main result

LitFM demonstrates state-of-the-art performance across literature-related tasks. As the authors state, "LitFM outperforms leading models (e.g., GPT-4o) across literature-related tasks, such as citation link prediction and paper recommendation." On the medicine dataset specifically, "LitFM achieves 8.0% improvement in accuracy for citation link prediction and 5.9% in Hits@1 for citation recommendation." The model also shows superior performance on related work generation, where "LitFM outperforms existing powerful LLMs on related work generation in both BERT and ROUGE scores," with improvements of "1.6% on the CS dataset and 9.8% on the Physics dataset" compared to GPT-4o.

Research paradigm

Design science / empirical validation

Author conclusions

The authors conclude: "In this paper, we develop the first literature foundation model LitFM that uniformly handles various citation graph-related tasks. It consists of a graph retriever and a knowledge-infused LLM to avoid hallucinations and generalize to handle new papers outside the citation graph. Experimental results demonstrate both the versatility of LitFM and its superior performance across six benchmark tasks." They also state that their work addresses the critical need for "a unified model capable of effectively performing diverse literature tasks-moving beyond a narrow focus on singular tasks-while mitigating hallucination issues, delivering high performance, and addressing the Matthew effect."

Risk of bias

Matthew effect inherent in citation graphs (bias toward highly-cited papers); Degree bias in message-passing mechanism favoring high-degree nodes; Dataset composition bias: CS (340k nodes), Physics (59k nodes), Medicine (2.1M nodes) - imbalanced domain representation; Potential selection bias in human evaluation study (participants were 'active researchers in the AI domain'); Training/test set split bias: connected dense subgraph sampling may not represent broader citation patterns; Dataset selection bias: citation graphs from arXiv and PubMed may not represent all domains; Matthew effect amplification: Initial tendency to recommend well-established papers over emerging research; Domain-specific sampling: Test set of 2,000 connected dense subgraphs may not be representative of overall graph distribution; Baseline selection: Some baselines (e.g., GAT with BERT embeddings) may not be optimally tuned

Limitations

The authors acknowledge: "However, one area where LitFM lags behind is English structure and coherence, with a score of 4.2, compared with 4.6 for DeepSeek-R1 and 5 for GPT-4o
This is due to the significantly larger number of parameters in GPT-4o and DeepSeek-R1, leading to advanced proficiency in English compared with LitFM." Additionally, the authors note that "current state-of-the-art (SOTA) graph-based retrievals face two key limitations: 1) Their performance highly depends on the precision and informativeness of given queries..
2) They rely solely on semantic similarity for retrieving relevant papers, which exacerbates the Matthew effect."

Open questions raised

The paper identifies gaps in existing approaches:
current graph-based retrievals lack robustness to incomplete or ambiguous queries, (2) they rely solely on semantic similarity without diversity consideration, exacerbating the Matthew effect, (3) single-task approaches limit knowledge transfer across diverse literature tasks, and (4) lack of citation context in existing datasets limits adaptability.
models that handle diverse literature-related tasks beyond single-task focus
retrieval methods that balance accuracy with diversity to mitigate the Matthew effect
unified approaches that internalize citation graph structure while maintaining knowledge transfer across tasks.
Most domain-specific models focus on single tasks, limiting cross-task knowledge transfer

Data: Three curated citation graph benchmarks from Medicine (PubMed Central), Computer Science (arXiv), and Physics (arXiv) - specific download links not provided in paper; Medicine dataset (PubMed Central repository): 2.1M nodes, 7.4M edges, 1.5M related works; Computer Science dataset (arXiv LaTeX sources): 340k nodes, 3.2M edges, 188k related works; Authors state: "We open-source LitFM, along with a graphical user interface, facilitating seamless model usage and enabling broader adoption in research and academic applications."Code: LitFM is open-sourced with a graphical user interface (specific GitHub/repository URL not provided in paper)Extracted from: pdf

Explore related topics

Systematic Review Automation Literature Discovery Design Science