AI evidence extraction

Topic Is Not Agenda: A Citation-Community Audit of Text Embeddings

Junseon Yoo · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Computational measurement study combining three components: (1) construction of a two-level citation-community partition on a 3.58M-paper augmented graph using Leiden CPM clustering at two resolutions (γ_L1=10^-4 for 73K sub-fields, γ_L2=10^-2 for 329K research agendas); (2) hierarchical nearest-neighbor analysis measuring same-community rate across four embedding models (Gemini, Qwen3-8B, Qwen3-0.6B, SPECTER2) on 1.6M standardized papers; (3) diagnostic retrieval evaluation on 80 curated research-agenda queries comparing seven retrievers (BM25, four cosine embeddings, citation-graph reranking, and BM25+citation hybrid) with manual validation of 16 representative queries..

Main result

The study found that "top-10 same-rate is 45-52% at L1 but only 15-21% at L2" across four state-of-the-art text embedding models, demonstrating that while embeddings successfully match papers at the sub-field (L1) level, they substantially fail at the research-agenda (L2) level. Additionally, "a deliberately simple citation-count rerank on top of plain BM25 candidates already reaches 59.6% top-1 L2, against 50.6% for the best cosine retriever (Gemini) and 39.3% for BM25 alone," showing that citation-graph structure carries significant signal that text embeddings miss.

Research paradigm

Empirical measurement of computational systems on real-world data

Author conclusions

The authors conclude that "Graph, community labels, queries, and code are released with this paper" and that "the citation-graph re-ranking step, not the choice of candidate generator" closes the agenda-level gap in retrieval. They emphasize that "citation-rerank wins where embeddings cannot rely on [distinctive vocabulary], which is the regime covering most of scientific literature." The paper demonstrates that "text embeddings are trained by co-occurrence; sub-field shares that surface content abundantly, which is why L1 match at 45-52% is non-trivial. Research agendas, in contrast, are defined by a conjunction of specifics the text does not always expose -instruments, mechanisms, drugs, organisms, methods—so two papers in the same L1 but different L2 look nearly identical to a text encoder."

Risk of bias

Selection bias in query construction: 10-of-25 query selection kept agendas with largest Boolean-search result sets, biasing benchmark toward well-populated agendas; Model family confound: Graph retriever uses Gemini Flash Lite 2.5 (same family as Gemini Embedding), though BM25+cite variant mitigates this; Domain representation bias: Citation-graph wins in domains where embeddings lack distinctive vocabulary; Physics and CS have embedded-favorable training-corpus bias; Sampling bias: Stratified sampling by publication year; standardization to 200K papers per domain may not represent all subfields equally; Corpus composition bias: Embeddings trained on modern web-scale corpora over-represent CS and Physics agenda vocabulary relative to Biomedical and Chemistry; Query selection bias: 10-of-25 queries kept those with largest Boolean-search result sets, biasing toward well-populated agendas; Model family confound: Graph retriever uses gemini-flash-lite 2.5 (same family as Gemini Embedding); Domain representativeness: Measurement covers scientific literature only; generalization to other reference-bearing corpora unverified; Corpus bias: CS and Physics agenda labels over-represented in modern embedding training corpora vs. specialist Biomedical/Chemistry vocabulary; Model family confound: Graph retriever uses Gemini Flash Lite (same family as Gemini Embedding) for query expansion and relevance judgment, though BM25+cite variant mitigates this; Corpus bias: CS and Physics domains show higher agenda matching due to over-representation in modern embedding training corpora, while Biomedical and Chemistry vocabulary is comparatively rare; Sampling bias: Stratified sampling by publication year and standardized per-domain downsampling may not equally represent all research threads within domains; Missing abstract bias: Small fraction of papers required crawling from publisher websites for missing abstracts

Limitations

The authors state that "(i) Citation communities are a proxy for research-agenda structure
our hard partition slightly over-counts disagreement for cross-agenda papers
(ii) The Graph retriever's query expander and relevance judge are both gemini-flash-lite 2.5, from the same model family as Gemini Embedding
the BM25+cite variant uses no LLM at the candidate stage, removes the confound, and reproduces the gain (59.6% top-1 L2), but a fully non-Gemini Graph configuration is still future work
(iii) Our 10-of-25 query selection kept the agendas with the largest Boolean-search result sets, biasing the benchmark toward well-populated agendas
(iv) The measurement covers scientific literature only

Open questions raised

Designing a learned reranker that uses the citation-graph signal (rather than the diagnostic simple count-based probe)
Verifying the agenda-granularity failure in other reference-graph-bearing corpora (legal opinions, patents, clinical guidelines, financial filings)
Fully non-Gemini Graph configuration for the retrieval pipeline
Generalizing the measurement beyond scientific literature to other domains with reference graphs
Fair score fusion strategy (per-query routing) to combine text embeddings and citation signals
Designing a learned reranker that uses citation-graph signal and verifying the effect in other reference graphs (legal, patents, clinical guidelines)

Data: Augmented citation graph and L1/L2 community labels: Zenodo, DOI 10.5281/zenodo.20046263 (CC-BY 4.0); 80 curated research-agenda queries and analysis code: https://github.com/junseon-yoo/topic-not-agenda; Per-(model, domain, k) same-community rates and enrichment ratios released as JSON files; leiden_cpm_sweep_v2_stats.json (resolution parameter sweep statistics); subfield_rerun_hier.json (per-query L1/L2 labels from manual validation); knn_hier_discordance.json (nearest-neighbor discourse validation data); 80 curated research-agenda queries with representative-paper IDs; leiden_cpm_sweep_v2_stats.json (per-γ cluster size statistics); subfield_rerun_hier.json and knn_hier_discordance.json (L2 validation data)Code: https://github.com/junseon-yoo/topic-not-agenda (MIT license); GitHub: https://github.com/junseon-yoo/topic-not-agenda; https://github.com/junseon-yoo/topic-not-agenda (Code and analysis scripts, MIT license)Extracted from: pdfAgreement 46%

Explore related topics

RAG for Research Empirical Study