AI evidence extraction

Chain of Retrieval: Multi-Aspect Iterative Search Expansion and Post-Order Search Aggregation for Full Paper Retrieval

Sangwoo Park, Jinheon Baek, Soyeong Jeong, Sung Ju Hwang · ArXiv.org · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2507.10057

Methodology & findings

Study design

Computational framework evaluation with empirical benchmarking.

Sample

N = 3200, 9 groups

Primary method

Reciprocal Rank Fusion (RRF) for merging ranked lists. Direct Preference Optimization (DPO) for training query optimizers. Recall@k and nDCG@k metrics for evaluation. Convex Hull Volume Ratio (CHV Ratio) for diversity analysis. Cohen's Kappa coefficient for inter-annotator agreement in human evaluation. Mean and standard deviation reported across three independent runs for ablation studies.

Main result

COR significantly outperforms existing retrieval baselines. Specifically, "COR outperforms all baselines across various settings, validating the effectiveness of our proposed framework for full paper-to-paper retrieval." The study found that "when using the same domain-agnostic retriever, COR surpasses abstract-to-abstract (A2A) baselines by an average of 6.37% in Recall, demonstrating that simply relying on abstracts is suboptimal compared to our aspect-driven approach."

Reports effect sizes and confidence intervals.

Research paradigm

Empiricist/Computational

Author conclusions

"In this work, we introduced Chain of Retrieval (in short, COR), a novel framework for paper-to-paper retrieval that iteratively expands the search space via aspect-aware query optimization (reinforced with DPO training) and recursively merges results via post-order aggregation. For evaluation, we presented SCIFULLBENCH, a large-scale dataset for full-context scientific retrieval, and results show that COR consistently outperforms the abstract-level and full-context baselines, even with off-the-shelf embedding models and for a patent retrieval task, highlighting its robustness and generality."

Risk of bias

Potential model contamination mitigated by using recent publications; citation-based relevance signals may reflect publication bias rather than true relevance; human evaluation relies on 15 participants with moderate inter-annotator agreement (Cohen's Kappa = 0.43); preference optimization depends on citation-based reward signals which may not capture all forms of relevance.; Model contamination potential due to use of recent publications from arXiv (2020-2025); Citation-based ground truth definition may not capture all relevant papers; Selection bias in venue choice (only major ML/NLP conferences: NeurIPS, ICLR, ACL, EMNLP); Potential model contamination in citation links used for relevance signals—mitigated by using recent publications (2020-2025); Reference section leakage—addressed through explicit removal of reference sections and citation markers from documents; Data leakage between training and evaluation—prevented by excluding query documents from training set that overlap with benchmark; Preference pair construction bias—uses citation-based rewards which may favor papers with strong citation signals over equally relevant uncited works

Limitations

"While our work introduces a novel retrieval approach that iteratively expands the search space via multi-aspect query optimization over the full context of scientific papers, it still has room for future work
First, an adaptive branch pruning mechanism could be further considered during the exploration phase to more dynamically guide the search results with the semantic alignment of individual papers with the root query, which would be an interesting direction for future work
Second, while our framework retrieves diverse and relevant papers, deeper exploration and repeated LLM inference introduce additional computational overhead."

Open questions raised

Future work directions include: (1) developing adaptive branch pruning mechanisms for more dynamic search guidance; (2) optimizing computational overhead from deeper exploration and repeated LLM inference; (3) exploring additional document domains beyond scientific papers and patents; (4) extending to multi-lingual retrieval scenarios.
Adaptive branch pruning mechanism during exploration phase to more dynamically guide search results based on semantic alignment
Optimization of computational overhead introduced by deeper exploration and repeated LLM inference
Further applicability to other long document-to-document retrieval tasks beyond scientific papers
Adaptive branch pruning mechanism for more dynamic search guidance during exploration phase
Computational optimization to reduce overhead from deeper exploration and repeated LLM inference

Data: SCIFULLBENCH - available at https://github.com/psw0021/Chain-of-Retrieval-Official; PATENTFULLBENCH - self-constructed benchmark for patent-to-patent retrieval (see Appendix D); SCIFULLBENCH; PATENTFULLBENCH; SCIFULLBENCH: Large-scale benchmark for full-context scientific paper retrieval, consisting of approximately 40K papers from arXiv (2020-2025) with 3200 query documents from ICLR, NeurIPS, ACL, and EMNLP venues (https://github.com/psw0021/Chain-of-Retrieval-Official); PATENTFULLBENCH: Self-constructed patent retrieval benchmark for evaluating generalization beyond scientific papersCode: https://github.com/psw0021/Chain-of-Retrieval-Official; GitHub: https://github.com/psw0021/Chain-of-Retrieval-OfficialExtracted from: pdfAgreement 53%

Explore related topics

AI-Assisted Discovery Literature Discovery Benchmarking