AI evidence extraction

ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure

A. J. Lew, Y. Cao, M. J. Buehler · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Computational benchmark design with systematic information disclosure levels.

Main result

The study found that "the more recent GPT-5.4 and Gemini 3.1 Pro Preview models consistently outperform their earlier counterparts across all levels of provided context." Additionally, "Performance generally improves with additional context. However augmenting the raw topic and research question with a null hypothesis yields a larger gain than further specifying an experimental procedure on top of that hypothesis, indicating diminishing marginal returns from increasingly detailed guidance." Furthermore, "GPT-5.4 maintains a relatively strong performance (F1 ≈ 0.70) even under minimal context, whereas the earlier Gemini 2.5 Pro model requires more complete experimental details to achieve comparable results."

Research paradigm

Empirical-analytical (positivist); computational evaluation framework

Author conclusions

The authors conclude that "Scientific discovery is inherently creative, uncertain, and forward-looking. Unlike recall-based benchmarks that evaluate known knowledge through exam-style questions, anticipating the outcomes of novel research -and assessing a model's ability to generate genuinely new insights -poses a fundamentally more challenging problem, requiring nuanced reasoning and deep domain understanding." They further state that "our approach evaluates how effectively large language models can reason toward plausible scientific results rather than simply retrieve them." Finally, they position the work as "an early step toward more rigorous evaluation of machine-assisted discovery, and as a guidepost for the development of systems that can more meaningfully accelerate scientific research."

Risk of bias

Judge model bias: Using GPT-5 as evaluator may favor GPT-family model outputs; Positional bias in claim alignment scoring; Dataset heterogeneity and difficulty variability across manuscripts; Domain imbalance in knowledge distribution of frontier models; Potential training contamination despite use of recent publications; Judge model bias: Using GPT-5 as evaluator for all claim comparisons, including for GPT-family models, may introduce systematic bias favoring certain phrasing styles; Training contamination risk: Although manuscripts from last 6 months were used to avoid training cutoff, model training dates and exact cutoff information not verified; Dataset representativeness: Only 45 manuscripts total (15 per category: bioactive, mechanical, nanomaterials) from Springer Nature, potentially biased toward certain material types; Domain imbalance: Bioactive manuscripts cluster near upper performance bound showing saturation, while mechanical manuscripts show wider variability and lower scores, suggesting unequal domain coverage; Positional bias: Although authors report flipping order of claims and averaging scores, potential residual positional effects remain; Judge model bias: Using GPT-5 as the evaluatory model for both GPT-family and non-GPT models introduces potential bias favoring GPT models; Positional bias: Partially mitigated by running alignment scoring twice with flipped claim order, though not eliminated; Selection bias: Limited dataset of 45 manuscripts from specific materials science domains may not represent broader scientific discovery; Training contamination risk: Reliance on recent papers (last 6 months) assumes models are not trained on these specific manuscripts, but cannot fully guarantee this; Domain imbalance: Heterogeneous difficulty across domains (bioactive materials show saturation effects while mechanical materials show wide performance distribution); Stochasticity in evaluation: Acknowledged limit of LLM stochasticity in validation tests affects score calibration

Limitations

The authors acknowledge that "using GPT-5 as the judge for GPT-family models may introduce bias" and state "we acknowledge a potential limitation in our current evaluation setup: using GPT-5 as the judge for GPT-family models may introduce bias
To address this, future iterations will incorporate cross-family evaluation using independent models." Additionally, the paper notes "the standard deviations across all scores are relatively large
This variability arises from the heterogeneous difficulty of the manuscripts included in the evaluation set." The dataset is limited in scope, using only "15 manuscripts per category for a total of 45 manuscripts" from materials science domains (bioactive materials, nanomaterials, and mechanical materials).

Open questions raised

Expansion of benchmark across broader scientific domains beyond materials science
Incorporation of wider range of models beyond GPT and Gemini families
Refinement of evaluation protocols through varying context granularity
Deeper analysis of reasoning processes in model projections
Cross-family evaluation using independent models to address judge bias
Systematic benchmarks for scientific discovery remain limited compared to factual recall benchmarks

Data: Open Access Springer Nature articles published within last 6 months (search terms: 'bioactive materials', 'nanomaterials', 'mechanical materials'); 45 manuscripts total (15 per category); Available at: https://github.com/〈future-link〉; Open Access Springer Nature articles published within last 6 months obtained via Springer Nature API. Dataset to be released at https://github.com/ (future-link). Table 4 in Appendix lists 45 manuscripts (15 per category: bioactive materials, nanomaterials, mechanical materials) with DOIs and publication dates from March 2026.; Dataset to be released at GitHub (future-link specified: https://github.com/〈future-link〉). Dataset consists of 45 manuscripts from Open Access Springer Nature API with search terms 'bioactive materials', 'nanomaterials', and 'mechanical materials' (15 per category), published within 6 months of submission. Specific manuscripts listed in Appendix Table 4 with DOIs and publication dates.Code: https://github.com/〈future-link〉 (benchmark will be released); https://github.com/ (future-link) - benchmark code repository planned for release but link not yet active at time of paper submission; https://github.com/〈future-link〉 (noted as future release)Extracted from: pdfAgreement 55%

Explore related topics

AI-Assisted Discovery Benchmarking