AI evidence extraction

Death of a reviewer or death of peer review integrity? the challenges of using AI tools in peer reviewing and the need to go beyond publishing policies

Vasiliki Mollaki · Research Ethics · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

5.41

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1177/17470161231224552

Methodology & findings

Study design

Narrative review and policy analysis examining the current landscape of AI tool usage in peer review processes and publishing policies.

Main result

The paper identifies that "there is a lack of guidelines or policies on whether peer reviewers can use such tools" as AI technology like ChatGPT becomes increasingly accessible. The authors argue that "we need to go beyond policies by creating transparent procedures that will enable journals to investigate allegations of non-compliance and take decisions that will protect the integrity of the peer review system."

Research paradigm

Critical interpretivism

Author conclusions

The authors conclude that "we need to go beyond policies by creating transparent procedures that will enable journals to investigate allegations of non-compliance and take decisions that will protect the integrity of the peer review system. Reviewers found to violate relevant policies must be excluded from the process to safeguard the integrity of the peer review system."

Risk of bias

As a narrative review/policy analysis (not an empirical study), traditional bias categories do not apply. However, potential selection bias in cited policies and potential framing bias in the authors' interpretation of peer review challenges may be present.

Open questions raised

The authors identify a critical gap: the absence of clear guidelines and policies governing AI tool usage specifically by peer reviewers in the peer review process, despite policies addressing AI use by manuscript authors.
The paper identifies a critical gap: the absence of clear guidelines and policies governing the use of AI tools (such as ChatGPT) by peer reviewers, despite the fact that publishing policies have addressed AI use by manuscript authors.
The paper identifies the need for transparent procedures enabling journals to investigate non-compliance with AI use policies in peer review, and the necessity to establish guidelines and policies specifically addressing peer reviewer use of AI tools.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 78%

Explore related topics

AI Peer Review Peer Review Position Paper