AI evidence extraction

Sense and sensibility of article submission platforms are needed regarding verification of AI use: a stakeholders’ perspective

Jaime A. Teixeira da Silva, Joshua Wang · AI and Ethics · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1007/s43681-025-00823-4

Methodology & findings

Study design

Systematic content analysis of article submission platforms (ASPs) and instructions for authors (IFAs) from top-ranked medical journals.

Sample

N = 50, 10 groups

Primary method

Descriptive content analysis with percentage calculations; systematic categorization of AI-related clauses in ASPs and IFAs; comparative analysis between ASP and IFA requirements; archival documentation via Internet Archive's Wayback Machine

Main result

The study found that "only 36% of the 36 journals' ASPs analysed in detail carried a clause related to AI, even though 90% of the same journals had ethics clauses related to AI use in their IFAs." Additionally, "of the 13 ASPs [with AI-related clauses], all 13 ASPs were a binary choice (Yes/No) regarding usage of AI, accompanied by a fairly simple query," and critically, "there is no reliable or replicable way to detect whether GAI was used to generate a passage of text with a level of certainty needed for editors to decide whether misconduct has taken place."

Reports effect sizes.

Research paradigm

Critical/Interpretive - examining structural inconsistencies in publishing systems and stakeholder perspectives on AI verification mechanisms

Author conclusions

The authors conclude that "To the best of our knowledge, this is the first study to examine the submission mechanisms, specifically via an ancillary tool, the ASP, through which journal editors attempt to monitor GAI use and enforce their policies. Our analysis revealed that these mechanisms (IFAs and ASPs) are misaligned in some top-ranked medical journals. Most of these journals mandate a declaration of GAI use in their IFAs, but do not formalize this declaration at the submission stage in their ASPs." They recommend that "editors are encouraged to personally and occasionally scrutinize the content of their journal's IFA to ensure that the requirements align with those specified on the journal's ASP."

Risk of bias

Selection bias: Only top 50 SJR-ranked medicine journals analyzed; findings may not generalize to lower-ranked or non-medical journals; Temporal bias: ASP and IFA analyses conducted at different time points (October and November 2024); platform content may have changed between analyses; Observation bias: First author's personal experience with two journal submissions influenced the study design and framing; Researcher interpretation: Subjective categorization of what constitutes 'AI-related clauses' may introduce inconsistency; Missing data: One journal (Current Protocols in Bioinformatics) excluded because IFA only available via pre-submission enquiry; Selection bias: Only top 50 SJR-ranked medical journals examined; may not represent lower-ranked or predatory journals; Temporal bias: Data collected in October-November 2024; ASPs and IFAs may change over time; Observer bias: Content analysis conducted by two authors with potential subjective interpretation; Exclusion bias: 14 journals excluded from Objective 1 analysis (13 invitation-only, 1 email submission); Selection bias: Only top 50 Q1-ranked medical journals examined; excludes lower-ranked and non-medical journals; Observation bias: First author had to create accounts for each journal, potentially affecting access or presentation; Temporal bias: Data collected October-November 2024; policies may change; Subjectivity: Qualitative interpretation of clause presence and consistency

Limitations

The authors note that "There is no conclusive evidence of the effectiveness of these declarations in deterring undeclared AI (GAI/LLM) use, as evidenced by the apparent absence of literature with quantitative data on AI declarations and author behaviour." Additionally, they acknowledge that "There is therefore an insufficient evidence base to determine how well ASP declarations actually deter authors from submitting undeclared material generated by AI, such as figures or images." The study is also limited by its focus on top 50 ranked journals only, and the authors acknowledge "Further analyses beyond the scope of this paper are needed to understand the roles and synergies of different declaration channels in holding academics to ethical standards at the article submission stage."

Open questions raised

Limited research scrutinizing ASPs in detail despite their fundamental role in academic publishing
Absence of literature with quantitative data on AI declarations and author behaviour
Insufficient evidence base to determine how well ASP declarations deter undeclared AI use
Unknown effectiveness of unverifiable pledges/declarations in preventing dishonest author behavior
Lack of consensus on which communication channels (ASP, cover letter, manuscript section) are best suited for integrity declarations
Need for further research on the roles and synergies of different declaration channels in enforcing ethical standards

Extracted from: pdfAgreement 61%

Explore related topics

Responsible AI Use Research Integrity Content Analysis