AI evidence extraction

Role of AI chatbots in education: systematic literature review

Lasha Labadze, Maya Grigolia · International Journal of Educational Technology in Higher Education · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

8/10

Relevance

2/4

Quality (LMQS)

Evidence

791

Citations

127.52

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s41239-023-00426-1

Methodology & findings

Study design

Systematic literature review following PRISMA-aligned methodology.

Sample

N = 67, 1 group

Primary method

Narrative synthesis and thematic analysis. The review employed qualitative synthesis of findings from reviewed studies. Authors explicitly note they "employed Large Language Model (LLM) for stylistic suggestions" in the discussion section. No formal meta-analytic techniques or quantitative pooling of effect sizes was performed.

Main result

The research findings emphasize the numerous benefits of integrating AI chatbots in education, as seen from both students' and educators' perspectives. We found that "students primarily gain from AI-powered chatbots in three key areas: homework and study assistance, a personalized learning experience, and the development of various skills." For educators, the main advantages are "the time-saving assistance and improved pedagogy." However, the review also emphasizes "significant challenges and critical factors that educators need to handle diligently. These include concerns related to AI applications such as reliability, accuracy, and ethical considerations."

Reports effect sizes.

Research paradigm

Interpretivist/qualitative synthesis

Author conclusions

The authors conclude that "students appreciate the capabilities of AI chatbots and find them helpful for their studies and skill development, recognizing that they complement human intelligence rather than replace it." They further state that "addressing some of the challenges related to the use of AI chatbots in education can be accomplished by introducing preventative measures. More specifically, educational institutions must prioritize creating awareness among students about the risks associated with AI chatbots, focusing on essential aspects like digital inequality and ethical considerations. Simultaneously, investing in the continuous development of educators through targeted training is key."

Risk of bias

Publication bias: Only peer-reviewed studies included; may miss gray literature or unpublished negative findings; Time-limited scope (2018-2023): Earlier or more recent research excluded; Language bias: Only English-language studies included; Selection bias: Authors excluded studies with 'limited empirical evidence' without explicit criteria for sufficiency; Reviewer bias: While multi-author review was used, no inter-rater reliability measures reported; Selection bias in article inclusion (peer-reviewed journals only, 2018-2023 timeframe); Publication bias (focus on published positive findings); Language bias (English-only papers); Potential bias in synthesis due to reliance on author interpretation of reviewed studies; Selection bias: Only English language peer-reviewed journals, books, and book chapters included; conference proceedings excluded; Publication bias: Review limited to 2018-2023 timeframe; earlier research excluded; Quality assessment bias: Authors conducted subjective quality checks based on research methodology, sample size, and research design clarity; Screening bias: Multiple authors reviewed articles, but potential for inconsistency despite inter-rater procedures; Publication bias: Only peer-reviewed journals, books, and book chapters were included; conference proceedings and grey literature excluded; Temporal bias: Limited to 2018-2023, excluding earlier foundational research; Selection bias: Authors acknowledge excluding studies with 'limited empirical evidence' at final stage, though criteria not explicitly defined; Heterogeneity in study designs: Systematic review of mixed study types without meta-analysis may obscure important contextual differences

Limitations

The authors note "There are a few aspects that appear to be missing from the existing literature reviews: (a) The existing findings focus on the immediate impact of chatbot usage on learning outcomes
Further research may delve into the enduring impacts of integrating chatbots in education, aiming to assess their sustainability and the persistence of the observed advantages over the long term
(b) The studies primarily discuss the impact of chatbots on learning outcomes as a whole, without delving into the potential variations based on student characteristics." Additionally, the paper acknowledges that "the field of chatbot development is continually emerging and requires timely, and updated analysis to ensure that the information and assessments reflect the most recent advancements, trends, or developments in chatbot technology."

Open questions raised

Long-term impacts of chatbot integration: Research has focused on immediate effects; sustainability and persistence of advantages over time need investigation.
Student characteristic variations: Impact variations based on age, prior knowledge, and learning styles not thoroughly explored.
Pedagogical mechanisms: Specific pedagogical strategies employed by chatbots and underlying instructional approaches need deeper investigation.
User experience and acceptance: Deeper research needed on usability, perceived usefulness, satisfaction, and preferences of students and teachers.
Contemporary chatbot capabilities: More research required on new-generation chatbots (ChatGPT, Bard) and their educational applications.
Long-term sustainability and persistence of chatbot advantages over time

Data: Authors state: "The data and materials used in this paper are available upon request. The comprehensive list of included studies, along with relevant data extracted from these studies, is available from the corresponding author upon request." No public repository URL provided.; The comprehensive list of included studies (67 articles) is available upon request from the corresponding author. No structured dataset repository provided.; The comprehensive list of included studies, along with relevant data extracted from these studies, is available from the corresponding author upon request.; "The data and materials used in this paper are available upon request. The comprehensive list of included studies, along with relevant data extracted from these studies, is available from the corresponding author upon request."Code: None mentioned.; No code repositories mentioned.Extracted from: pdfAgreement 44%

Explore related topics

Systematic Review Automation Knowledge Synthesis Literature Review