AI evidence extraction

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

Sy-Tuyen Ho, Minghui Liu, Huy Nghiem, Furong Huang · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Benchmark construction from 35,209 ICLR submissions and 137,940 expert reviews, yielding 1,099 validated hypothesis-experiment pairs.

Sample

N = 1099, 2 groups

Primary method

Confusion matrix analysis (binary classification with categorical metrics: true positives, true negatives, false positives, false negatives). Reported metrics: recall (True Positive Rate per class), Macro F1 (unweighted average of per-class F1 scores). No frequentist hypothesis tests or interval estimation reported. Evaluation setup: max_tokens=8192, temperature=0.2 across models; BM25 retrieval with k=3 for atomic-claim verification; threshold τ=0.7 for support ratio.

Main result

The study found that "under standard prompting, models frequently classify low-soundness proposals as high-soundness, yielding a mean false-positive rate of 74.0%" and "Under the tested aggressive prompt, models over-correct: while the false-positive rate drops to 19.9%, high-soundness recall collapses to 36.1%." These results demonstrate "a pervasive optimism bias against reviewer-derived proposal-stage labels" and indicate that "current frontier models are not yet reliable as standalone first-gate critics for ML research proposals."

Reports effect sizes.

Research paradigm

Empirical-positivist; benchmarking and measurement-based evaluation of LLM performance on a constructed task

Author conclusions

The authors conclude: "Across the models we tested, judgments are often sensitive to prompt framing: under standard prompting many models over-approve weak proposals, while stricter prompting can push some models toward broad over-rejection." They further state: "Reliable proposal-stage scientific judgment will likely require targeted training, calibration, or human-in-the-loop review beyond prompting alone." Additionally, "SoundnessBench is an imperfect but audited proxy for recoverable proposal-stage soundness, and the observed optimism bias is unlikely to be explained solely by leakage, memorization, simple style features, or a narrow topic slice."

Risk of bias

Label noise: Reviewer soundness scores are imperfect proxies; reviewers saw full papers while models see results-masked proposals; Public-corpus contamination: ICLR submissions are public; evaluated models may have encountered source papers during pretraining; Extraction drift: Proposal extraction from PDFs could introduce errors despite verification procedures; Prompt sensitivity: Model judgments are highly sensitive to prompt framing variations; Selection bias in filtering: Only papers with high reviewer agreement (mean confidence ≥3, std <15%) were retained; Desk-rejection exclusion: Papers desk-rejected were removed, potentially biasing the remaining sample; Label noise: Reviewer soundness sub-scores are imperfect proxies; only high-agreement reviews retained (confidence ≥3/5, std<15%); Public-corpus contamination: ICLR papers are public; model pretraining may have encountered source papers; Extraction fidelity: LLM extraction may introduce drift; mitigated by near-verbatim preservation and atomic-claim auditing; Reviewer expertise variation: Reviewers saw full papers and results, while models see results-masked proposals only; Domain-specific bias: Benchmark limited to ICLR ML research; generalizability to other venues/domains unclear; Outcome leakage: Preliminary human audit found 92.3% of leakage checks matched expected 'No' answer but 7.7% may leak; Label validity: 84.6% of label-validity checks matched expected 'Yes' in preliminary audit; 15.4% ambiguous; Prompt sensitivity: Models show high sensitivity to prompting regime; results conditional on prompt framing; Label noise from reviewer subjectivity and mixed soundness scores; Public corpus contamination risk from ICLR submissions in training data; Extraction drift and information leakage from proposal-text processing; Reviewer agreement filtering may exclude hard-to-judge cases; Model memorization of recognizable method names or paper identities; Prompt sensitivity introducing framing bias; Instruction-tuning and RLHF encouraging model agreeableness

Limitations

The authors state: "Our ground truth relies on reviewer soundness sub-scores, which are expert signals but still imperfect proxies for proposal-only methodological validity because reviewers saw full papers, results, presentation quality, and framing
SoundnessBench should therefore be interpreted as measuring recoverable pre-execution soundness signals rather than exact full-paper review prediction or definitive post-execution research quality." Additionally, "The benchmark also covers a bounded slice of ML research drawn from ICLR
extending to other venues and scientific domains such as biology, chemistry, and social science is important before making claims about scientific soundness in general
Because the source corpus is public, perfect contamination control is impossible." Further, "our human audit is preliminary and does not establish a full expert-human ceiling."

Open questions raised

Extending SoundnessBench to other venues beyond ICLR and scientific domains such as biology, chemistry, and social science
Expanding human expert re-annotation and establishing a full expert-human ceiling
Developing private or continuously refreshed test sets to reduce contamination concerns
Incorporating richer modalities (e.g., code and logs) beyond proposal text
Conducting longitudinal proposal-to-execution studies to validate proposal-stage judgments against actual research outcomes
Investigating whether targeted training on scientific judgment can resolve the optimism bias

Data: SoundnessBench: 1,099 validated ML research proposals from ICLR 2022-2026 (to be released); SoundnessBench: 1,099 validated ML research proposals extracted from ICLR corpus (2022-2026), including 458 low-soundness and 641 high-soundness instances. Source: ICLR public corpus with reviewer metadata. Availability status in paper: not explicitly stated.; SoundnessBench: 1,099 validated ML research proposals derived from ICLR 2022-2026 submissions (dataset release status not explicitly confirmed in paper, though described as constructed)Code: Not mentionedExtracted from: pdfAgreement 52%

Explore related topics

AI Peer Review Peer Review Benchmarking