AI evidence extraction

Publishers’ and journals’ instructions to authors on use of generative artificial intelligence in academic and scientific publishing: bibliometric analysis

Conner Ganjavi, Michael Eppler, Asli Pekcan, Brett M. Biedermann, Andre Luis Abreu, Gary S. Collins et al. · BMJ · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

199

Citations

20.07

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1136/bmj-2023-077192

Methodology & findings

Study design

Cross-sectional bibliometric study involving manual systematic search of websites of the top 100 largest academic publishers (identified by number of affiliated journals) and top 100 highest ranked scientific journals (identified by H-index from Scimago) screened on 19-20 May 2023 with updated search on 8-9 October 2023.

Sample

N = 200, 2 groups

Primary method

Descriptive statistics were used to summarize the data, reporting frequencies and percentages for all categorical variables. No inferential statistical tests were performed. Data presentation included charts and tables to aid interpretability.

Main result

Among the top 100 largest publishers, 24% provided guidance on the use of GAI, of which 15 (63%) were among the top 25 publishers. Among the top 100 highly ranked journals, 87% provided guidance on GAI. Of the publishers and journals with guidelines, "the inclusion of GAI as an author was prohibited in 96% and 98%, respectively." Only one journal (1%) explicitly prohibited the use of GAI in the generation of a manuscript. Where to disclose the use of GAI varied, including in the methods or acknowledgments, in the cover letter, or in a new section. "GAI guidelines in 12 journals directly conflicted with those developed by the publishers."

Reports effect sizes.

Research paradigm

Positivist/Empiricist

Author conclusions

"Substantial heterogeneity was found in guidance on the application of GAI use in academic research and scholarly writing." The authors conclude that "none of the proposed recommendations were formulated through a structured consensus based guideline development process." They state that "this scenario highlights an urgent need for the establishment of cohesive, cross disciplinary policies. Such guidance should be crafted in a structured manner, integrating the perspectives of all stakeholders. This approach is crucial to counteract the Babel Tower phenomenon—that is, the confusion and lack of standardization that results from individual parties creating their own unique regulations."

Risk of bias

Selection bias: Study limited to top 100 publishers and top 100 journals; other publishers/journals not captured; Proxy measurement bias: Use of subsidiary journal guidelines as proxies for publisher policies may not accurately represent publisher position; Subjective interpretation bias: Study noted as qualitative and prone to authors' subjectivity, though mitigated by multiple reviewers; Temporal bias: Data collected at two time points only (May and October 2023); rapid evolution of guidelines could be missed; Language bias: Non-English websites translated using Google Machine Translate, which may introduce inaccuracies; Snapshot bias: Guidelines continue to evolve; findings represent specific moments in time rather than stable state; Selection bias: Limited to top 100 publishers by journal count and top 100 journals by H-index; may not represent smaller or emerging publishers/journals; Measurement bias: Qualitative assessment by human reviewers prone to subjectivity despite multiple reviewer protocol; Language bias: Non-English guidelines translated using Google Machine Translate, which may introduce translation errors; Temporal snapshot bias: Data collected at two discrete time points (May and October 2023), not continuous monitoring; Proxy bias: Use of subsidiary journal websites as proxies for publisher policies when publisher websites lacked explicit guidance; Selection bias: Limited to top 100 publishers and top 100 journals by specific metrics; Information bias: Manual website screening subject to interpretation; non-English websites translated using Google Machine Translate; Temporal bias: Rapid evolution of guidelines during study period (25% increase in journal guidelines between searches); Subjectivity bias: Acknowledged as largely qualitative study prone to authors' subjectivity, though mitigated by multiple reviewers

Limitations

This study was "a snapshot at six months and 10 months after the rise in popularity of ChatGPT." The authors state that "weaknesses of the current study include the limited number of publishers and journals included." They also note that "some publishers lacked policies on their websites, and the shared subsidiary journal guidelines that we used as proxies may not always be the perfect solution for published guidelines." Additionally, "this study was largely qualitative and therefore prone to authors' subjectivity," though the authors attempted to minimize this through structured multiple reviewers.

Open questions raised

Need for standardized, cross-disciplinary guidelines on GAI use in academic publishing
Need for formal, structured consensus-based guideline development processes using established methodologies
Follow-up study needed to evaluate the full effect of scholarly societies on GAI guideline development, as their role was not fully captured in this study
Future guidelines may need to be discipline-specific as understanding of GAI technology improves
Need for standardized, cross-disciplinary GAI guidelines developed through formal consensus-based guideline development processes
Evaluation of scholarly society guidelines' impact on GAI use recommendations (requires follow-up study due to limited representation in top 100 journals)

Data: Full data available from corresponding author (Giovanni.cacciamani@med.usc.edu) upon requestExtracted from: pdfAgreement 62%

Explore related topics

Institutional Policy AI Governance Content Analysis