AI evidence extraction

TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews

Hanqi Duan, Xiang Li · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Design science research with benchmark construction, artifact development, and experimental evaluation.

Primary method

Design science with tool decomposition based on evidence type; iterative refinement informed by ablation studies and distribution-shift evaluation

Main result

TADDLE achieves strong binary detection performance with "Accuracy (86.00%), Precision (91.54%), and F1 (0.8230), exceeding the strongest baseline, RottenReviews-LR (F1 0.8018), by 2.12 F1 points." On multi-label classification, "TADDLE is best on all four metrics, with Jaccard 0.7424, Micro F1 0.4828, Macro F1 0.4332, and Weighted F1 0.5009." Under cross-conference and cross-generator distribution shifts, "TADDLE obtains the highest ∆ in all four columns (ICML: 0.453; NeurIPS: 0.418; Llama 3.3-70B: 0.427; MiniMax-M2.5: 0.281)."

Research paradigm

Design science, empirical evaluation with benchmarking

Author conclusions

"We studied the problem of detecting deficient LLM-generated reviews and proposed TADDLE, a tool-augmented agent-based system that reframes detection as an explicit analysis–verification–decision pipeline. TADDLE employs an orchestrator to route review segments through specialized analysis tools, and a fine-tuned integration module to synthesize the analyses into final decisions. Extensive experiments on a newly constructed benchmark demonstrate that TADDLE significantly outperforms strong baselines and maintains its advantage under multiple distribution shifts."

Risk of bias

Distribution bias: Review distribution shaped by six generator LLMs and nine persona prompts rather than naturally occurring AI-assisted reviews; Annotation bias: Annotators may rely on subtle persona-related artifacts when assigning labels; Generator selection bias: Test set uses DeepSeek V3.2 (held out from training), but other generators in training may not be representative; Paper selection bias: High-disagreement papers deliberately selected (score difference ≥3 for ICLR, ≥2 for ICML/NeurIPS) rather than random sampling; Venue bias: Benchmark limited to three ML venues (ICLR, ICML, NeurIPS); generalization to other disciplines unclear; Persona coverage bias: Six deficient personas designed to ensure categorical coverage rather than reflect realistic defect distributions; Persona-conditioned artifact: reviews generated under explicit defect personas may not reflect organic LLM failure modes; Annotator bias: domain experts may unconsciously rely on persona artifacts when assigning labels despite labeling against review text only; Distribution shift: benchmark reflects 2025 LLM capabilities and may not generalize to future models; Venue scope: limited to ML conferences (ICLR, ICML, NeurIPS); unclear transfer to other disciplines; Generator selection: only six LLM generators used; results may be generator-specific; Class imbalance: rare defect categories (BIAS, CARELESS, LACK_CONSTRUCTIVENESS) have limited support (<10% in test set, as low as 3 positives per category); Test-set construction: balanced test distribution (68.89% non-deficient) may not reflect realistic review populations; Persona-based review generation may introduce systematic artifacts that annotators rely on when assigning labels; Synthetic review distribution may not reflect organic AI-assisted reviewing patterns; Limited to three ML venues (ICLR, ICML, NeurIPS); potential discipline-specific bias; Expert annotators may be influenced by subtle persona artifacts despite guidelines; High-disagreement paper sampling creates selection bias favoring challenging cases

Limitations

The authors identify three major limitations
First, "Our benchmark consists of LLM-generated reviews that are multi-label-annotated by domain experts against the original review text
While the labels are grounded in human judgment, the review distribution is shaped by six generator LLMs and nine persona prompts
The personas are intended as a coverage mechanism rather than a realistic model of organic AI-assisted reviewing, and annotators may also rely on subtle persona-related artifacts when assigning labels." Second, "The defect taxonomy is derived from LLM-generated reviews from 2025
as LLM capabilities evolve, new deficiency patterns may emerge that are not covered by the current six categories
Likewise, the benchmark covers three ML venues

Open questions raised

Validation on real-world AI-assisted reviews: authors note "More broadly, organically produced AI-assisted reviews may differ systematically from persona-conditioned ones. We therefore view this benchmark as an initial step and leave validation on real-world AI-assisted reviews to future work."
Taxonomy evolution: new deficiency patterns may emerge as LLM capabilities evolve
Cross-discipline generalization: whether the taxonomy transfers to other disciplines (medical, legal, social science peer review) is an open question
Class imbalance handling: authors note rare classes (BIAS, CARELESS, LACK_CONSTRUCTIVE) appear in fewer than 10% of reviews, suggesting "class reweighting, loss functions emphasizing rare classes, and richer rare-class augmentation are natural next steps"
Validation on real-world organically produced AI-assisted reviews (currently only synthetic persona-conditioned reviews)
Transfer of defect taxonomy to other disciplines beyond ML (medical, legal, social science peer review)

Data: TADDLE benchmark: 1,800 expert-annotated reviews on 50 ICLR 2025 papers. Available at https://github.com/AquariusAQ/TADDLE; TADDLE benchmark: 1,800 expert-annotated reviews on 50 ICLR 2025 papers (40 training papers, 10 test papers); https://github.com/AquariusAQ/TADDLE; TADDLE benchmark: 1,800 expert-annotated reviews on 50 ICLR 2025 papers (https://github.com/AquariusAQ/TADDLE); Cross-conference evaluation sets: ~900 reviews each from ICML 2025 and NeurIPS 2025 papers; Cross-generator evaluation sets: ~900 reviews each from Llama 3.3-70B and MiniMax-M2.5Code: https://github.com/AquariusAQ/TADDLEExtracted from: pdfAgreement 62%

Explore related topics

AI Peer Review Peer Review Design Science