AI evidence extraction

Computational Support in Academic Peer Review: An Artificial Intelligence Perspective

Riswahyuni Widhawati, Suryari Purnama, Herman Purwoko Putro, Lumi Gantari, Shakeel Rahagi · ADI Journal on Recent Innovation (AJRI) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.34306/ajri.v6i1.1106

Methodology & findings

Study design

Literature review combined with experimental approach using machine learning algorithms (Natural Language Processing, Supervised Learning algorithms including Random Forest and Support Vector Machines, and Unsupervised Learning algorithms such as K-Means Clustering) tested on a dataset of approximately 500 manuscripts from computer science, engineering, and social sciences..

Sample

N = 500, 3 groups

Primary method

The paper describes using machine learning algorithms (Natural Language Processing, Random Forest, Support Vector Machines, K-Means Clustering) for analysis, but no formal statistical tests, p-values, or confidence intervals are reported. The paper states evaluation metrics would be employed to compare AI-driven model with traditional peer review, but specific statistical methods and results are not detailed.

Main result

The study found that "the model demonstrated a high level of accuracy in identifying weaknesses and strengths within the manuscripts, successfully recognizing methodological errors, data inconsistencies, and other critical issues" and that "the implementation of AI led to substantial improvements in time efficiency, allowing peer reviews to be completed in a matter of hours rather than days."

Reports effect sizes.

Research paradigm

positivist

Author conclusions

The authors conclude that "the findings of this research highlight the transformative potential of artificial intelligence in the academic peer review process. The model's ability to accurately identify weaknesses and strengths, coupled with significant improvements in review efficiency, provides a compelling case for the broader adoption of AI technologies in scholarly publishing. By addressing the challenges associated with traditional peer review, AI can play a critical role in advancing the quality and integrity of scientific research."

Risk of bias

Potential for bias in AI models; Data privacy concerns; Risk of over-reliance on automated systems undermining human reviewer role; No discussion of reviewer selection bias for human comparison group; Limited detail on dataset composition and representativeness; Potential bias in AI models; Risk of over-reliance on automated systems; Data privacy concerns not fully addressed; Potential for bias in AI models (acknowledged by authors); No explicit blinding of reviewers mentioned; Dataset composition not fully detailed (mix of published/unpublished manuscripts from computer science, engineering, social sciences); No discussion of inter-rater reliability or validation against independent human reviewers; No control group or baseline comparison explicitly described; Selection bias in manuscript dataset not addressed

Limitations

The authors identify several limitations: "several challenges must be addressed, including the potential for bias in AI models, data privacy concerns, and the risk of over-reliance on automated systems, which could undermine the essential role of human reviewers."

Open questions raised

Enhancing algorithm transparency
Developing bias detection and correction mechanisms
Establishing ethical guidelines for AI usage in peer review
Future research should focus on responsible application of AI in peer review
Need to address challenges of potential bias in AI models, data privacy concerns, and over-reliance on automated systems
Ensuring responsible application of AI in peer review systems

Data: Dataset comprises approximately 500 manuscripts from multiple journals in computer science, engineering, and social sciences - no public repository URL provided; Approximately 500 manuscripts from computer science, engineering, and social sciences journals (published and unpublished) - specific URLs or access information not providedExtracted from: pdfAgreement 68%

Explore related topics

AI Peer Review Peer Review Conceptual