AI evidence extraction

ACL-Verbatim: hallucination-free question answering for research

Gábor Recski, Szilveszter Tóth, Nadia Verdha, István Boros, Ádám Kovács · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Computational/simulation approach combining: (1) automated synthetic query generation using the ScIRGen methodology paired with document retrieval; (2) human annotation by NLP researchers of 100 query-chunk pairs from the ACL Anthology; (3) training of extractive models on silver supervision; (4) evaluation using word-level, containment, and coverage metrics on the manually annotated benchmark..

Main result

The study found that "a 150M-parameter ModernBERT token classifier trained on silver supervision from our pipeline achieves the best word-level F1 (53.6), ahead of the strongest evaluated LLM extractor (48.7)." The ACL-specialized ModernBERT model also achieves the highest word-level precision and "often abstains on irrelevant chunks," demonstrating improved precision for extraction-based question answering.

Research paradigm

Empirical/Computational

Author conclusions

"We described an application of the VerbatimRAG architecture to over 100K research papers in the ACL Anthology, contributed a manually annotated dataset for the core extraction task, and presented a set of experiments showing that small customized encoder-decoder architecture trained with synthetic data outperforms zero-shot LLM-based extraction on this task, at a fraction of the cost. We release all components of our pipeline as open-source software. We believe that combining the VerbatimRAG approach with the task-oriented training of extractive models provides a blueprint for the efficient deployment of high-performing hallucination-free question answering systems across a variety of domains."

Risk of bias

Limited annotator pool (NLP researchers only, no inter-annotator agreement measurement); Small manually annotated dataset (100 query-chunk pairs) may not represent diverse query types; Potential bias reproduction from LLM-generated silver supervision used for training; Cherry-picked examples used to illustrate model behavior; Domain-specific annotation requiring expertise in parsing algorithms and NLP; Small manually annotated dataset (100 query-chunk pairs); Potential bias from synthetic training data generated by LLMs; Limited inter-annotator agreement measurement; Dataset limited to ACL Anthology papers (computational linguistics domain); Annotation performed by authors rather than independent annotators; Small manually annotated dataset (100 query-chunk pairs only) limits generalizability; Limited inter-annotator agreement measurement due to complexity of annotation task; Lack of rigorous adjudication process for resolving annotator disagreements; Potential reproduction of LLM biases in synthetic training data; Cherry-picked examples used for qualitative analysis; Annotation performed by authors (potential confirmation bias)

Limitations

"The validity of our conclusions is limited by the size of the manually annotated dataset that was the basis of both quantitative and qualitative evaluation
The high complexity of the annotation task, described in detail in Section 3.2, also limited our ability to measure agreement between multiple annotators, to implement a rigorous adjudication process for resolving differences among annotators, or to develop detailed and objective annotation guidelines." Additionally, "the extraction models trained using synthetic training data may reproduce unintended bias present in the output of LLMs, which may lead to such bias being reinforced and propagated by our models."

Open questions raised

The authors suggest that similar datasets should be created for "more narrowly defined question answering use-cases that in turn lead to more objective extraction tasks," particularly for domain-specific applications where expert annotation is more feasible. They note the need for better handling of domain-specific expertise in academic question answering.
Need for more narrowly defined question answering use-cases with more objective extraction tasks
Potential for extension to other domains beyond academic research
Domain-specific expertise required for reliable annotation in specialized fields
Measure inter-annotator agreement and develop detailed annotation guidelines
Need for ground truth datasets mapping user queries to relevant text spans in research papers

Data: ACL Anthology markdown files (114,475 files) - Released on HuggingFace under CC-BY 4.0 license at https://huggingface.co/datasets/KRLabsOrg/acl-anthology-md; Manually annotated benchmark dataset (100 query-chunk pairs) - Available in acl-verbatim repository; Silver supervision dataset (23,235 rows, based on 2000 sampled papers with 5,892 synthetic queries) - Part of verbatim-spans dataset; ACL Anthology markdown corpus; Manually annotated benchmark; verbatim-spans dataset; Silver supervision dataset; ACL Anthology markdown files; Manually annotated gold benchmark; verbatim-spans multi-domain datasetCode: ACL-Verbatim: https://github.com/KRLabsOrg/acl-verbatim; VerbatimRAG framework (referenced, not directly released in this paper); ACL Anthology: https://github.com/acl-org/acl-anthology/; https://github.com/KRLabsOrg/acl-verbatim; ACL-Verbatim; VerbatimRAGExtracted from: pdfAgreement 51%

Explore related topics

RAG for Research Literature Discovery Empirical Study