AI evidence extraction

Does Artificial Intelligence Advance Science?

Liangping Ding, Cornelia Lawson, Philip Shapira · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

3/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Large-scale bibliometric analysis of over 1 million publications from OpenAlex (2004-2024).

Sample

N = 1127716, 2 groups

Primary method

Probit regression models with average marginal effects (AMEs). Robust (heteroskedasticity-robust) standard errors estimated using delta method. Publication year fixed effects and OECD field fixed effects included in all models. Stratified random sampling (2% of non-AI publications, stratified by field) used to construct comparison group. Two-sample t-tests used for mean difference comparisons in Table 2. Field-specific novelty thresholds calculated within mapped OECD field classifications.

Main result

The study found that "AI publications are significantly more likely to achieve top-decile creativity relative to non-AI publications, with 5.5 to 10.2 percentage point higher likelihood to rank in the top creativity decile." Additionally, "Tool-oriented AI research, which applies existing AI models to domain tasks, is associated with the largest gains in recombinant-based creativity, while Adaptation-oriented AI research, modifying AI models for domain-specific problems, is associated with relatively higher object-based creativity."

Reports effect sizes.

Research paradigm

Positivist empiricism with quantitative bibliometric analysis

Author conclusions

The authors conclude: "These findings reveal that AI does not advance science through a single mechanism but through structurally distinct creative pathways that depend on how AI is incorporated into the research process." They further state: "AI's relationship with scientific creativity is unlikely to be homogeneous. Instead, creative outcomes are contingent on how AI is integrated into the knowledge production process, influencing the mechanisms through which novelty and impact emerge." Additionally: "current AI primarily functions as a technology that expands combinatorial search within existing knowledge structures when used as a tool in research, rather than one that generates fundamentally new conceptual objects. However, we show that this combinatorial advantage is not uniform across research contexts."

Risk of bias

Selection bias from using published research only (peer review and editorial filtering effects); Classification bias from GPT-SciBERT pipeline potentially misclassifying AI-relevant publications; Measurement bias: indirect creativity measures cannot capture paradigm shifts or tacit knowledge creation; Temporal bias: citation windows (3-year, 10-year) may not capture delayed recognition ('sleeping beauties'); Field heterogeneity in citation and novelty practices controlled via OECD field fixed effects; Publication type differences (journals vs. conference proceedings) controlled but may still confound; Attrition from missing data (reference information for recombinant novelty, phrase data for object novelty); Selection bias: Study excludes preprints and non-peer-reviewed outputs, which are overrepresented in AI research, potentially underestimating AI's impact; Classification bias: AI relevance determined through keyword matching and semantic embeddings; potential false positives/negatives in mode classification despite filtering pipeline; Publication bias: Analysis restricted to published research; reflects editorial and peer review preferences that may favor AI publications; Data quality bias: OpenAlex records may have incomplete citation and reference information, particularly for newer publications; Confounding: Multiple unobserved factors may drive both AI adoption and creativity (researcher quality, institutional prestige, field dynamics); Temporal bias: Recent publications have shorter observation windows for citation accumulation, potentially underestimating long-run impact for post-2022 papers; Measurement bias: Citation counts and novelty metrics are indirect proxies for creativity; may not capture paradigm shifts or tacit knowledge; Selection bias: Sample restricted to peer-reviewed articles and conference proceedings, excluding preprints which are disproportionately represented in AI research; Classification bias: Automated keyword and semantic filtering may miss AI adoption or misclassify Discussion-mode papers; Temporal bias: Missing values in 10-year citations due to discrepancy between SciSciNet and OpenAlex version; Missing data bias: Recombinant novelty scores missing for papers without complete reference information; Publication bias: Results reflect peer review and editorial preferences, not ground truth of scientific contributions; Evaluation system bias: Citation metrics reflect evaluation system preferences rather than pure scientific merit

Limitations

The authors state: "We recognize several important limitations in the measurement approach
Our analysis relies on publications as the unit of analysis, which cannot capture informal knowledge creation occurring through unpublished work, knowledge exchange in collaboration, or tacit learning" and "our results pertain to AI as a scientific instrument and epistemic technology, not as background support for scholarly writing or idea generation
Our creativity measures are necessarily indirect and capture only observable novelty and impact
While recombinant and object novelty reflect important aspects of scientific creativity, they cannot fully represent deeper conceptual shifts, paradigm changes, or tacit knowledge creation." Additionally, "because our analysis relies on published research, it reflects both knowledge production and its selection through peer review and editorial processes."

Open questions raised

Heterogeneity in AI effects across individual researchers (productivity gains, output quality vary by skill level and experience)
Whether AI amplifies or narrows disparities in scientific creativity across worker characteristics and experience levels
Effects of AI on creativity in organizational contexts and team structures
Longer time horizons needed to capture paradigm changes and conceptual shifts
AI's influence on serendipitous recombination and distant search in knowledge space
Heterogeneity across workers: 'Understanding whether AI ultimately amplifies or narrows disparities in scientific creativity therefore remains an important open question. Future research should examine how the effects of AI on creativity vary across worker characteristics, experience levels, and organizational contexts'

Data: OpenAlex; SciSciNet-v2; Arts et al. (2025) object novelty dataset; OpenAlex (https://openalex.org) - primary data source containing over 450 million bibliometric records. Snapshot retrieved May 30, 2025.; SciSciNet-v2 - used for pre-calculated fixed-window citation impact data; OpenAlex (https://openalex.org) - snapshot retrieved May 30, 2025; SciSciNet-v2 version - used for pre-calculated fixed window citation impactCode: None mentioned. No GitHub or code repository links provided.Extracted from: pdfAgreement 53%

Explore related topics

Productivity Effects Research Productivity Empirical Study