AI evidence extraction

Can ChatGPT be an author? A study of artificial intelligence authorship policies in top academic journals

Brady Lund, K.T. Naheem · Learned Publishing · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

2.90

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1002/leap.1582

Methodology & findings

Study design

Content analysis of AI authorship policies from the top 300 academic journals based on ScimagoJR indexing factor as of April 1, 2023.

Sample

N = 300, 6 groups

Primary method

Content analysis using NVivo software for sentiment analysis (scale: -1.0 to 1.0) and term frequency analysis. Manual tallying and categorization of policy content. Descriptive statistics (frequencies and percentages) reported for policy characteristics. Chi-square or similar comparisons not explicitly reported.

Main result

Of the 300 journals examined on April 1, 2023, 176 of them (58.7%) had a specific policy posted online that pertained to the use of AI tools like ChatGPT. Of these 176 policies, 170 (96.6%) allowed for using ChatGPT to improve the quality of manuscripts, while six policies (3.4%) prohibited any use of these models. Nearly all policies (98.9%) explicitly mention that these tools should NOT be included in the authorship list, whereas just two policies (1.1%) do not specifically prohibit the inclusion of an AI tool like ChatGPT as an author.

Reports effect sizes.

Research paradigm

Positivist/empiricist - systematic content analysis of published policies

Author conclusions

The authors conclude that "As AI tools continue to transform all aspects of our daily lives, it is important that our policies and procedures adapt to ensure the quality and integrity of our disciplines. This study reveals that many top academic journals are actively working to address the possible pitfalls and issues that may emerge in this new academic reality." They further state "it is essential to shift the responsibility for these policies from reviewers and editorial assistants to editors and editorial boards, in collaboration with major academic publishers."

Risk of bias

Selection bias: Journals ranked by ScimagoJR may not represent full diversity of academic publishing landscape; Temporal bias: Data collected at single time point (April 1, 2023); policies likely changed afterward; Publication bias toward high-impact journals: Sample limited to top 300 by indexing factor; Assumption that online-posted policies are representative of actual practice; Manual coding of policy content introduces potential coder bias despite collaborative approach; Temporal bias - data collected at specific cutoff (April 1, 2023) with rapid policy changes likely; Selection bias - only top 300 journals by ScimagoJR indexing examined, not representative of all journals; Publisher concentration - findings affected by small number of major publishers controlling many top journals; Manual coding bias - subjective determination of journal disciplinary focus and policy content classification; Temporal limitation: data collection at single time point (April 1, 2023), with high likelihood of policy changes occurring between data collection and publication; Selection bias: examined only top 300 journals by ScimagoJR indexing factor, potentially excluding smaller or emerging journals; Classification bias: disciplinary classification of journals performed by researchers without clearly stated validation procedures

Limitations

The authors note that "it is not only possible, but probable, that some AI authorship policies have been updated in the time between data collection and publication of this manuscript." Additionally, they acknowledge that "while it might be beneficial to examine policies at the publisher level, as some publishers have established AI policies that apply to all their journals, researchers usually choose publication venues based on factors like journal prestige, rather than the publisher itself." The study employed a fixed data collection date (April 1, 2023), making findings time-sensitive.

Open questions raised

The authors identify the need for discipline-specific analysis, noting that "further analysis of the AI authorship policies of journals within these disciplines is still needed" for physics, journalism, and medical sciences. They also suggest that future research can serve "as a point of comparison to the findings of this study" as policies continue to evolve. The authors note inconsistencies in publisher approaches, highlighting "the necessity for ongoing discussions and exploration within the publishing industry to develop standardized practices."
Further discipline-specific analysis needed, particularly for physics, journalism, and medical sciences
Need for ongoing discussions to develop standardized practices across the publishing industry
Inconsistencies among journals from the same publisher regarding where AI acknowledgements should appear
Future research needed to compare evolving policies over time
Further analysis of AI authorship policies within specific disciplines (physics, journalism, medical sciences) is needed

Data: List of 300 journals examinedExtracted from: pdfAgreement 63%

Explore related topics

Institutional Policy AI Governance Content Analysis