AI evidence extraction

The Role of Artificial Intelligence in the Lifecycle of Scientific Manuscripts: Authoring, Reviewing, and Editorial Selection

Jose L. Domingo · Qeios · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

3/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.32388/9aob4v.2

Methodology & findings

Study design

Critical commentary integrating analysis of empirical studies (2023-2025) on AI citation fabrication, detection capabilities, and peer review applications.

Primary method

The paper does not employ original statistical methods. It synthesizes and critically analyzes empirical findings from prior studies (2023-2025) on AI hallucination rates, accuracy metrics, and categorical comparisons of human vs. AI capabilities.

Main result

The paper finds that AI tools demonstrate significant hallucination risks in academic publishing. The authors report that "while GPT-4 improved upon GPT-3.5's 55% fabrication rate to 18%, subsequent testing of GPT-4o showed it fabricated approximately 20% of citations while introducing errors in 45% of real references." Additionally, "a 2025 cross-model study by Cabezas-Clavijo and Sidorenko-Bautista revealed that only 26.5% of AI-generated references were entirely correct, with nearly 40% being erroneous or completely fabricated." However, AI shows promise as a technical auditor: "AI can detect inconsistencies in p-values, degrees of freedom, and mathematical derivations with over 90% accuracy."

Reports effect sizes.

Research paradigm

Critical commentary and policy analysis rooted in empirical evidence review

Author conclusions

The authors conclude that "the integration of AI into scientific publishing is not a simple question of adoption. It is a complex negotiation at a time of systemic crisis" and that "the only ethically and scientifically defensible path is a model akin to Scenario 3: The Fair Hybrid Model." They further argue that "this requires viewing AI not as a replacement, but as a tool that handles technical arduous work, thereby making the valuable time of human experts more efficient and sustainable. It must be paired with fair compensation for those experts, reforming the economic model that has brought us to this crisis point."

Risk of bias

Normative framing: The paper advances a clearly normative argument for specific policy measures, which may introduce interpretive bias in how evidence is selected and presented; Study selection bias: As a narrative review, the choice of which empirical studies to cite is not driven by systematic search criteria; Geographic bias in source literature: The paper notes that cited empirical studies reveal 'geographic and economic biases' in AI hallucination rates, particularly affecting lower-income countries; Publication bias in cited studies: The paper does not assess publication bias in the empirical studies it reviews; Selection bias in cited studies: focus on high-profile LLM models (GPT-3.5, GPT-4, ChatGPT, Claude); Potential geographic bias in AI training data as evidenced by studies showing higher error rates in low-income countries; Confirmation bias: the normative framework may predispose interpretation toward emphasizing AI risks over benefits; Publication bias: discussion relies on available empirical studies without systematic search strategy to identify all relevant evidence; Selection bias in cited empirical studies - heterogeneous methodologies and sample sizes; Publication bias in favor of studies demonstrating AI failures/hallucinations; Author's normative stance explicitly favors human-centered approaches and may bias interpretation of evidence; Limited representation of AI developer perspectives; Geographic bias in AI model testing - studies primarily focus on English-language models

Limitations

The paper explicitly states it is "presented as a critical commentary and policy analysis rather than a systematic review," indicating that comprehensive systematic synthesis was not attempted
Additionally, the paper acknowledges that "the solution space remains open" and notes uncertainty about which governance models will prove most viable, stating "other models, including those maintaining the traditional uncompensated contribution of referees within a hybrid AI-assisted system, may also offer viable paths forward depending on community values and economic feasibility."

Open questions raised

The paper identifies that the solution space for AI integration remains open, noting 'other models, including those maintaining the traditional uncompensated contribution of referees within a hybrid AI-assisted system, may also offer viable paths forward depending on community values and economic feasibility.' It calls for compulsory AI literacy programs for scientists and the need for robust legal frameworks assigning accountability when AI-assisted reviews fail.
Need for comprehensive AI literacy programs in academic institutions to train scientists on AI capabilities, limitations, and verification protocols
Insufficient legal frameworks and accountability mechanisms when AI-assisted reviews fail
Lack of standardized verification protocols for detecting AI-hallucinated citations across disciplines
Missing empirical studies on the effectiveness of hybrid peer-review models
Limited research on the long-term impact of algorithmic editorial selection on scientific diversity and innovation

Extracted from: pdfAgreement 59%

Explore related topics

AI Peer Review Peer Review Literature Review