12,637 papers · updated 18 Sept 2026livingmeta.ai
← Browse all papers
AI evidence extraction

Beyond Paper-to-Paper: Structured Profiling and Rubric Scoring for Paper-Reviewer Matching

Yicheng Pan, Zhiyuan Ning, Ludi Wang, Yi Du · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10
Relevance
D
Evidence
0
Citations
0.00
FWCI

Methodology & findings

Study design

Computational framework design with empirical evaluation on three public benchmarks.

Primary method

Design science research with iterative component validation. The design follows a coarse-to-fine pipeline approach combining multiple computational stages, with ablation studies validating each component's necessity.

Main result

P2R achieves state-of-the-art performance on reviewer matching benchmarks, with "P2R establishes a new state-of-the-art, securing the top rank in 10 out of 13 metrics and ranking second in the remaining three. This superiority is consistent across diverse domains, including machine learning (NeurIPS), information retrieval (SIGIR), and the broad scientific landscape (SciRepEval)." On the NeurIPS dataset specifically, "P2R achieves a substantial gain of +5.14 percentage points in Hard P@5 compared to the strongest baseline."

Research paradigm

computational design science

Author conclusions

"P2R advances reviewer matching by directly modeling paper–reviewer fit with multi-dimensional signals rather than relying solely on paper representations. Beyond improved accuracy, our framework offers a practical recipe for applying LLMs to reviewer matching: use LLMs to build structured expertise profiles, retrieve candidates with high recall, and enforce decision criteria via rubrics during final scoring."

Risk of bias

Model-specific bias: reliance on Qwen3 for profiling may introduce model-dependent biases; Limited dataset representation: evaluation restricted to three conference benchmarks; Annotation bias: ground truth labels derived from Area Chair decisions, which may reflect their subjective preferences; Sparse annotation: limited number of ground-truth reviewers per submission; LLM hallucination risk: despite JSON-only prompting constraints, potential for model-generated inaccuracies in profile extraction; Reliance on expert-curated labels which may reflect annotator biases; No discussion of potential fairness issues in reviewer recommendation; Stochasticity in LLM outputs - authors report mean performance over three runs to mitigate; Hyperparameter choices (RRF weights, discretization thresholds) not fully justified or sensitivity-tested; Training-free approach claims may mask implicit biases from pre-trained LLM embeddings

Open questions raised

  • The authors identify that prior paper-to-paper matching methods implicitly represent reviewers through publication history and rely on textual similarity alone, which is insufficient for capturing multi-dimensional expertise. They note that structural mismatches (e.g., shared topics but differing methodologies or application domains) are difficult to resolve by aggregating paper similarities, motivating the shift toward explicit multi-dimensional characterization.
  • The paper identifies that existing paper-to-paper matching paradigms are insufficient because "effective reviewer matching requires capturing multi-dimensional expertise, and textual similarity to past papers alone is often insufficient." Future work is implied to explore applications of this framework beyond the three tested conference domains.
Data: NeurIPS benchmark (34 papers, 190 reviewers, 393 annotated pairs); SIGIR benchmark (73 papers, 189 reviewers, 13,797 annotated pairs); SciRepEval benchmark (107 papers, 661 reviewers, 1,729 annotated pairs)Code: https://github.com/kg4sci/P2RExtracted from: pdf

Explore related topics

AI Peer ReviewPeer ReviewDesign Science

Related papers

← Browse all papers