AI evidence extraction

Peer Review in the Age of Generative AI

Atreyi Kankanhalli · Journal of the Association for Information Systems · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

3.83

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.17705/1jais.00865

Methodology & findings

Study design

Opinion piece analyzing the potential role of AI in peer review processes through conceptual discussion rather than empirical measurement or experimentation.

Main result

The paper identifies that "there is a pressing need to use AI to augment the review process" while simultaneously noting that "advances in AI have been accompanied by concerns about biases introduced by AI tools and the ethics of using them, among other issues such as hallucinations." The author argues that critical issues to understand are "how can AI augment and potentially automate the review process, what are the pitfalls in doing so, and what are the implications for IS research and peer review practice."

Reports effect sizes.

Research paradigm

Interpretive/argumentative

Author conclusions

The author offers "views on these issues in this opinion piece" regarding "how can AI augment and potentially automate the review process, what are the pitfalls in doing so, and what are the implications for IS research and peer review practice," emphasizing the tension between the need to use AI to address high submission volumes and reviewer scarcity, and the concerns about AI biases, ethics, and hallucinations.

Risk of bias

Author perspective bias (opinion piece format); No empirical validation of claims; Potential confirmation bias in selecting which AI concerns to emphasize

Limitations

The abstract notes that "we lack an in-depth understanding of how AI can impact the peer review process," indicating acknowledged gaps in current knowledge
The author presents this as "an opinion piece" rather than a comprehensive empirical analysis, which inherently limits the scope to conceptual exploration rather than validated findings.

Open questions raised

The paper identifies a critical gap: "we lack an in-depth understanding of how AI can impact the peer review process" and emphasizes the need to understand the implications of AI use for peer review practice in IS research.
The paper identifies that "we lack an in-depth understanding of how AI can impact the peer review process" and highlights the need to understand pitfalls in using AI for peer review, as well as implications for IS research and peer review practice.
The paper identifies a pressing need for in-depth understanding of AI's impact on peer review processes. It highlights gaps in knowledge about how AI can augment and automate review while managing pitfalls related to AI biases, ethics, and hallucinations in the context of IS research publishing.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 76%

Explore related topics

AI Peer Review Peer Review Position Paper