AI evidence extraction

HalluCiteChecker: A Lightweight Toolkit for Hallucinated Citation Detection and Verification in the Era of AI Scientists

Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Design science approach with artifact development and performance evaluation.

Primary method

Design science research with modular component-based architecture following five design principles: Easy to Use, Lightweight, Offline, Non-Generative AI, and Simple Implementation.

Main result

The toolkit successfully detects hallucinated citations in papers from EMNLP 2025, demonstrating practical utility. "HALLUCITECHECKER successfully detects hallucinated citations in papers from the EMNLP 2025 Main track reported in Sakai et al. (2026). This indicates that it achieves at least comparable recall to that of prior analyses." Processing performance testing on diverse hardware shows the toolkit operates efficiently: on a MacBook Pro, the system processes papers at 7220.9ms per paper (141.6ms per citation), while on a MacBook Air it requires 16361.7ms per paper (320.8ms per citation), and on WSL it takes 34666.5ms per paper (679.7ms per citation).

Research paradigm

Design science / Engineering

Author conclusions

The authors conclude: "By releasing both the task formulation and implementation, we aim to support further research in citation verification and scientific reliability." They further state: "If tools such as HALLUCITECHECKER become widely adopted within the community, similar to existing tools such as ACL PUBCHECK, we encourage community-driven maintenance and governance to ensure transparency, sustainability, and alignment with community needs. We hope this work contributes to establishing fair, transparent, and responsible scientific practices."

Risk of bias

Limited evaluation scope primarily to ACL venues; lack of independent benchmarks for hallucinated citation detection; reliance on fuzzy matching with fixed threshold (0.9) may introduce false positives/negatives; no human evaluation of false positive rates reported.; Limited evaluation scope (primarily ACL venues); Absence of comprehensive benchmark datasets; Reliance on pre-trained models (BidLSTM_CRF_FEATURES with GloVe embeddings) may have inherent biases; Fuzzy matching threshold (0.9) is arbitrary and not validated across diverse citation formats

Limitations

The authors acknowledge several limitations: "Although concurrent work has begun to explore hallucinated citation detection, many existing tools and analyses are not easily reproducible, are environment-specific, or are not designed for general-purpose use across diverse publication formats." Additionally, "This work primarily evaluates hallucinated citation detection using papers from ACL venues, but the toolkit is designed to generalize to other conferences such as ICLR and NeurIPS
However, unlike ACL Anthology, which maintains versioned archives of published papers, some venues overwrite manuscripts with updated versions without preserving historical revisions." The paper also notes that "comprehensive benchmarks for hallucinated citation detection remain limited."

Open questions raised

The authors identify several gaps: (1) "comprehensive benchmarks for hallucinated citation detection remain limited"; (2) "many existing tools and analyses are not easily reproducible, are environment-specific, or are not designed for general-purpose use across diverse publication formats"; (3) lack of versioned archival practices across many venues; (4) need for future dataset construction and benchmark development for hallucinated citation detection.
Limited comprehensive benchmarks for hallucinated citation detection
Lack of reproducible and environment-agnostic tools for general-purpose use across diverse publication formats
Need for dataset construction and benchmark development for hallucinated citation detection
Importance of transparent and versioned archival practices to support reproducibility and verification
Future improvements in recognition accuracy and efficiency through parsing, OCR, and post-correction

Data: Bibliographic databases derived from ACL Anthology, arXiv, and DBLP are included as default settings. The toolkit distributes resources through Hugging Face Hub (glove840B embeddings in LMDB format) and Hugging Face Datasets (citation databases using Apache Arrow).; ACL Anthology; arXiv; DBLP; ACL Anthology (via Hugging Face Datasets); arXiv (via Hugging Face Datasets); DBLP (via Hugging Face Datasets)Code: The toolkit is distributed as a Python package via PyPI (pip install hallucitechecker). A YouTube video demonstration is referenced but the repository URL is not explicitly provided in the paper.; PyPI (pip install hallucitechecker); Hugging Face Hub (for pretrained weights and citation databases); PyPI package: hallucitechecker; ACL PUBCHECK (referenced as similar tool): https://github.com/acl-org/aclpubcheckExtracted from: pdfAgreement 60%

Explore related topics

Hallucination Control Research Integrity Design Science