AI evidence extraction

What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature

Chung Kwan Lo · Education Sciences · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

1/4

Quality (LMQS)

Evidence

1,725

Citations

61.20

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.3390/educsci13040410

Methodology & findings

Study design

Rapid review following PRISMA statement.

Main result

The findings of this review suggest that "ChatGPT's performance varied across subject domains, ranging from outstanding (e.g., economics) and satisfactory (e.g., programming) to unsatisfactory (e.g., mathematics)." Additionally, "Although ChatGPT has the potential to serve as an assistant for instructors (e.g., to generate course materials and provide suggestions) and a virtual tutor for students (e.g., to answer questions and facilitate collaboration), there were challenges associated with its use (e.g., generating incorrect or fake information and bypassing plagiarism detectors)."

Research paradigm

Interpretivist/Qualitative synthesis

Author conclusions

"This rapid review of 50 articles highlighted ChatGPT's varied performance across different subject domains and its potential benefits when serving as an assistant for instructors and as a virtual tutor for students. However, its use raises various concerns, such as its generation of incorrect or fake information and the threat it poses to academic integrity. The findings of this review call for immediate action by schools and universities to update their guidelines and policies for academic integrity and plagiarism prevention. Furthermore, instructors should be trained on how to use ChatGPT effectively and detect student plagiarism. Students should also be educated on the use and limitations of ChatGPT and its potential impact on academic integrity."

Risk of bias

Selection bias: 64% of included articles (32 of 50) were preprints that had not undergone rigorous peer review; Geographic bias: Majority of articles from Western contexts (United States N=19, United Kingdom N=4, Austria N=3); Subject domain bias: Overrepresentation of medical education studies relative to other domains (mathematics, language education underrepresented); Publication bias: Articles retrieved may not represent full landscape of research (many mass media and social media articles excluded); Search strategy limitations: Limited by availability of publications within 3-month window after ChatGPT release; Majority of included articles (64%) were preprints without rigorous peer review; Geographic bias toward Western context (19 from US, 4 from UK); Disciplinary bias toward medical education and higher education contexts; Limited articles on primary/secondary education and non-Western contexts; Publication bias toward early ChatGPT adopter studies; Publication bias toward Western/English-language articles; Selection bias: majority of included articles (64%) were preprints without peer review; Context bias: concentration on medical and higher education domains; Temporal bias: limited to first three months of ChatGPT release (December 2022-February 2023); Small sample sizes in some included studies limiting generalizability

Limitations

"The majority of the included articles were preprints, meaning that they have not undergone rigorous peer review
The quality of their evidence is, therefore, questionable." Additionally, "most of the included articles were written in the Western context, particularly in medical and higher education
Thus, the findings of this review may be biased towards these specific contexts." Furthermore, "this review only focused on the original release of ChatGPT
As a result, the findings might not be applicable to other applications of GPT and GPT-4 that have been launched beyond the time period of this review."

Open questions raised

Very few studies empirically examined the influence of ChatGPT on student performance and behavior; research with larger sample sizes needed
Need for more rigorous, evidence-based studies rather than suggestions based on intuitive beliefs
Limited research in non-Western educational contexts
Underrepresentation of primary and secondary education contexts (most studies in higher education)
Limited research in certain subject domains (mathematics, language education)
Need for studies on newer versions (GPT-4) and other GPT applications

Extracted from: pdfAgreement 82%

Explore related topics

Responsible AI Use Academic Writing Literature Review