AI evidence extraction

AI Research Agents Narrow Scientific Exploration

Yixuan Tang, Yi Yang · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Computational simulation and empirical analysis.

Main result

The study found that "AI-generated ideas within the same research area are substantially more similar to one another (0.82-0.84) than human-authored papers from those same areas (0.77)" and that "AI-generated ideas remain substantially closer to the seed literature (0.92) than later human follow-on papers do (0.88)". Additionally, "papers most similar to AI-generated ideas receive 50.4 citations on average, compared with 54.9 citations for the same-area baseline" with a mean difference of -4.47 citations (95% CI: [-6.41, -2.53], p < 0.001). The analysis also revealed that "in 85.1% of AI-generated ideas, the research question already appears in the seed literature", while only 62.6% contain technical methods already in the seed literature.

Research paradigm

Empirical-analytical; computational methodology with quantitative analysis of AI-generated scientific ideas

Author conclusions

The authors conclude that "current AI research agents appear better suited to local elaboration than exploration. They efficiently recombine and refine ideas around existing conceptual neighborhoods, but less frequently move toward the more dispersed directions later explored by human researchers." Furthermore, they state that "the central question may not only be whether AI systems can generate coherent scientific ideas, but whether they can help expand the range of scientific directions under consideration. As AI research agents become more deeply integrated into scientific workflows, designing agentic AI systems that support exploratory breadth may become increasingly important."

Risk of bias

Selection bias: Analysis limited to papers from only three major ML conferences (ICLR, NeurIPS, ICML), not representative of broader scientific landscape; Measurement bias: Semantic similarity thresholds (0.87, 0.9) are manually calibrated and may not capture all meaningful research distinctions; Confounding: Comparison between AI-generated and human-authored papers may not account for differences in publication review processes, time of emergence, and peer feedback mechanisms; Temporal bias: Analysis uses papers published 2022-2025 for generation but citation data may be incomplete for recent papers; Model bias: Different LLMs exhibit substantially different validity rates (32.3% for Llama-3.2-1B to 99.9% for Gemma-4-31B-IT), which could bias results toward higher-capacity models; Framework bias: ResearchAgent achieves highest completion rate while AgentLaboratory shows lower validity due to multi-agent complexity; Prompt bias: All frameworks include explicit instructions to generate novel ideas, which may bias toward certain solution patterns; Selection bias: Research areas limited to 19 active citation-defined clusters from only three AI/ML conferences (ICLR, NeurIPS, ICML); may not represent broader scientific landscape.; Methodological bias: Reliance on embedding-based semantic similarity (cosine distance) to define research areas and match ideas; arbitrary threshold choices (0.87 for method/question matching) may bias results.; Confounding: AI agents explicitly instructed to propose novel, high-impact ideas; instructions may not fully propagate due to prompt engineering variability and LLM differences.; Temporal confounding: Citation patterns of human papers collected retrospectively; follow-on papers may be subject to publication delays and citation accumulation biases.; Model selection bias: Evaluation limited to four specific agent frameworks; other AI research agent designs not evaluated.; LLM representation bias: Evaluation of six LLMs, but the paper does not account for potential systematic biases in how different models respond to novelty instructions.; Semantic embedding bias: The choice of embedding model (Qwen3-Embedding-4B) and similarity threshold (0.87) may introduce systematic bias in measuring novelty and idea similarity.; Citation proxy bias: Using citation counts of semantically similar human papers as a proxy for AI-generated idea impact may not accurately reflect actual scientific value or future impact.; Literature domain bias: Analysis restricted to ICLR, NeurIPS, and ICML conferences (AI/ML focused), limiting generalizability to other scientific disciplines.; Seed paper selection bias: Bootstrap sampling of five seed papers may systematically bias what types of ideas are generated based on initial literature context.; Framework representation bias: Different agent frameworks produce outputs in heterogeneous formats; standardization to common research question/method representation may introduce biases.

Limitations

The authors note that their analysis relies on "matching human-authored papers semantically similar to generated ideas at a similarity threshold of 0.9" which "provides an exploratory measurement of whether regions close to AI-generated ideas are associated with higher downstream scientific impact." They acknowledge limitations of this proxy approach, stating that "because AI-generated ideas themselves do not have real-world citation outcomes, we instead identify human-authored papers that are semantically similar to the generated ideas and use their citation patterns as a proxy for downstream scientific impact." Additionally, the paper notes that "the similarity threshold of 0.87 is manually calibrated through inspection of representative semantic matches and mismatches" which may introduce calibration subjectivity
The analysis is further limited to papers from three top-tier ML conferences (ICLR, NeurIPS, ICML) rather than broader scientific domains.

Open questions raised

Need for AI research agent designs that support exploratory breadth rather than local elaboration
Investigation of mechanisms by which AI agents could generate fundamentally new research questions rather than recombining existing methods
Extension of analysis beyond machine learning/AI conferences to broader scientific domains
Study of how to integrate AI agents into scientific workflows while maintaining exploratory diversity
The authors identify that existing evaluations of AI research agents focus on whether individual ideas are interesting, novel, feasible, or executable, but reveal much less about how repeated AI-assisted ideation shapes the broader landscape of scientific exploration. They highlight the need to design agentic AI systems that support exploratory breadth rather than solely local elaboration, and suggest that future work should address whether AI can help expand the range of scientific directions under consideration rather than concentrate exploration in narrow conceptual neighborhoods.
The authors identify that 'the ability to generate research ideas at scale does not necessarily imply broader scientific exploration' and call for future research on how to design AI systems that support exploratory breadth in scientific discovery. They note that 'the central question may not only be whether AI systems can generate coherent scientific ideas, but whether they can help expand the range of scientific directions under consideration.'

Data: Citation network and paper metadata from DBLP: 34,698 papers from ICLR, NeurIPS, and ICML (2019-2025) with abstracts, publication years, and citation links; The authors constructed a dataset containing 34,698 papers from ICLR, NeurIPS, and ICML conferences published between 2019-2025, collected from DBLP with associated metadata (titles, authors, abstracts, keywords, citation links). The dataset is referenced but no explicit repository URL or availability statement is provided in the paper.; DBLP dataset containing 34,698 papers from ICLR, NeurIPS, and ICML (2019-2025) with abstracts, publication years, and citation links. The paper notes "with citation information from DBLP." No explicit availability statement or download link provided.Extracted from: pdfAgreement 54%

Explore related topics

AI-Assisted Discovery Research Productivity Empirical Study