AI evidence extraction

Human versus artificial intelligence: investigating ability of young academics from research and non-research institutions to identify ChatGPT-generated dental research abstracts

Matheel AL-Rawas, Omar Abdul Jabbar Abdul Qader, Galvin Sim Siang Lin, Yew Hin Beh, Muhammad Annurdin Sabarudin, Yee Ang et al. · Scientific Reports · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1038/s41598-026-42555-3

Methodology & findings

Study design

Cross-sectional observational study with six young academic reviewers blinded to abstract origin.

Sample

N = 150, 7 groups

Primary method

Chi-Square analysis (non-parametric), Spearman's correlation coefficient (bivariate correlation), crosstabulation, sensitivity/specificity/accuracy calculations using MedCalc Software Ltd. evaluation calculator (version 2023). Statistical significance level established at 0.05. Sample size calculation using G*power version 3.1.9.6 software based on significance level (α) of 0.05, power of 0.8, and effect size of 0.3.

Main result

The study found that "the overall accuracy of human reviewers in detecting abstract origin varied from 44% to 76%" while "GPTZero was the most accurate AI-output detector, with an accuracy rate of 90%" and "The similarity detector was 94% accurate." The research also revealed that "institutional setting appeared to be an inconsistent predictor of detection capability at an early career stage, while individual differences (e.g., prior exposure to research writing, reviewing experience, and familiarity with AI-assisted writing) may play a larger role."

Reports effect sizes.

Research paradigm

Positivist/empiricist

Author conclusions

The authors conclude: "it is evident at present that ChatGPT and its related tools and technologies will persistently impact the field of education. Emphasizing continuous research to mitigate the possible hazards will [incomplete in text]" and further state that "both types of universities need to offer structured training in AI literacy" and institutions "should not depend on a single detector; instead, they should adopt a layered strategy: AI literacy training for young dental academicians and the combined use of detection tools with human oversight for interpretation and ethical judgment."

Risk of bias

Small sample of reviewers (n=6) limiting generalizability; Selection bias in recruitment of young academics (random sampling from specific universities only); Potential observer bias despite blinding measures; Limited to dental research abstracts only (domain specificity); ChatGPT 3.5 version used; newer versions released after study completion; No explicit blinding confirmation mechanism described; Unequal distribution of detection capability across individual reviewers despite equal training; Small sample size of 6 reviewers may limit generalizability; Selection bias: reviewers were randomly selected but limited to those with <2 years experience from specific institutions; Potential reviewer bias due to subjective rubric-based assessment; ChatGPT version 3.5 used but newer versions released after study completion; Lack of inter-rater reliability testing among the six reviewers; Small sample size of reviewers (n=6); Potential selection bias in recruitment of young academics from specific universities; Temporal limitation: study used ChatGPT 3.5; newer versions released after study completion; Limited to dental field abstracts, reducing generalizability; Blinding of reviewers does not eliminate cognitive biases in abstract evaluation

Limitations

The authors stated that "The limited number of academicians used to review the abstracts is a limitation of the present study
A newer version of ChatGPT was released after the study was completed
Another limitation is that Turnitin® had not released their AI writing detection tool at the time of writing."

Open questions raised

The authors identify that no prior dental study has "focused specifically on early-career dental academicians across different institutional settings (research vs non-research) while benchmarking their performance against multiple AI-output detectors and similarity detection." They emphasize the need for "focused institutional initiatives that integrate AI literacy (critical evaluation of AI-assisted writing, disclosure standards, and ethical principles) with practical verification processes and mentorship-driven research skills enhancement" in dental education.
The study identifies the need for: (1) focused institutional initiatives that integrate AI literacy with practical verification processes and mentorship-driven research skills enhancement in dental education; (2) dental-specific guidance on generative AI use; (3) structured training in AI literacy for young academics; (4) further research on early-career academics' ability to detect AI-generated content across institutional settings
The authors identify that no prior dental study had focused specifically on early-career dental academicians across different institutional settings (research vs non-research) while benchmarking their performance against multiple AI-output detectors and similarity detection. They also note the "scarcity of dental-specific guidance and studies" regarding generative AI and call for structured institutional initiatives integrating AI literacy with practical verification processes.

Data: Not explicitly stated; paper does not mention datasets available for public access.Code: Not mentioned in the paper.Extracted from: pdfAgreement 59%

Explore related topics

Authorship & Attribution Research Integrity Experimental