AI evidence extraction

What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education

Ahmed Tlili, Boulus Shehata, Michael Agyemang Adarkwah, Aras Bozkurt, Daniel T. Hickey, Ronghuai Huang et al. · Smart Learning Environments · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

5/10

Relevance

3/4

Quality (LMQS)

Evidence

1,587

Citations

56.17

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s40561-023-00237-x

Methodology & findings

Study design

Qualitative instrumental case study conducted in three stages: (1) social network analysis of 2,330 tweets from 1,530 Twitter users collected December 23, 2022 to January 6, 2023; (2) content analysis of semi-structured interviews with 19 early adopters of ChatGPT (educators, students, developers, AI freelancers) who had publicly posted experiences with ChatGPT; (3) hands-on user experience testing by three experienced educators using ChatGPT across ten educational scenarios over one week with daily meetings..

Sample

N = 1552, 7 groups

Primary method

Social Network Analysis (SNA) using Harel-Koren Fast Multiscale algorithm; Sentiment analysis (binary classification: positive/negative/non-categorized); t-SNE (t-distributed stochastic neighbor embedding) for dimensionality reduction and visualization; Content analysis following steps by Erlingsson and Brysiewicz (2017) with two independent coders using predefined coding scheme; Betweenness centrality values for node sizing; Edge weight analysis for network visualization.

Main result

The study found that "the positive sentiments (5%) outweigh the negative sentiments (2.5%)" in social media discourse about ChatGPT in education, and "the investigation of user experiences through ten educational scenarios revealed various issues, including cheating, honesty and truthfulness of ChatGPT, privacy misleading, and manipulation." Additionally, "most participants felt the humaneness of ChatGPT needs to be improved, especially in terms of enhancing its social role."

Reports effect sizes.

Research paradigm

Interpretivist/constructivist with empirical qualitative components

Author conclusions

"This study followed a three-stage instrumental case study, namely social network analysis of tweets, content analysis of interviews, and investigation of user experiences, to examine the concerns of using chatbots in education, among early adopters, through the study of using ChatGPT. The obtained results revealed that while ChatGPT is a powerful tool in education, it still needs to be used with more caution, and more guidelines about how to use it safely in education should be established." The authors further conclude that "the findings of this study provide several research directions that should be considered to ensure a safe and responsible adoption of chatbots, specifically ChatGPT, in education."

Risk of bias

Selection bias: Study focused on early adopters only, not representative of broader population; Language bias: Twitter data limited to English tweets only; Temporal bias: Cross-sectional tweet analysis limited to specific 2-week period (Dec 23, 2022 - Jan 6, 2023); Query bias: Different search queries could yield different SNA results; Small sample size for interviews (n=19) and user experience testing (n=3 educators); Selection bias: Early adopters may have different perspectives than general population of educators/students; Self-selection bias: Interview participants were recruited from those who publicly posted about ChatGPT experiences; Language bias: Twitter analysis limited to English-language tweets only; Temporal bias: Cross-sectional tweet analysis limited to specific 2-week period in early 2023; Small sample size for interviews (n=19) and user testing (n=3); Sentiment analysis methodology not fully detailed, potential for coding bias; Interview familiarity rating average of 3.02/5 indicates mixed expertise levels; Selection bias: Study focused on early adopters only, potentially overrepresenting enthusiastic users; Temporal bias: Cross-sectional tweet analysis limited to specific 2-week period (December 23, 2022 - January 6, 2023); Sampling bias: Limited number of interview participants (n=19) and educators (n=3); Search query bias: Different search strings could produce different results in social network analysis; Self-selection bias: Interview participants were recruited from those publicly posting about ChatGPT experiences

Limitations

"This study mainly focused on early adopters of ChatGPT in education
It also relied on qualitative analysis without the use of quantitative analysis
Particularly, SNA provides a cross-sectional perspective and the tweets are limited to a specific time period including Tweets in English
Additionally, SNA with different search queries might lead to different results
Moreover, the number of participants involved in this study was limited (19 interviewees and 3 educators)."

Open questions raised

Needed competencies to effectively use and manage chatbots - what competencies are needed and how are they developed?
Development of humanized chatbots with reflective thinking and emotional capacity
Design of responsible chatbots beyond privacy and security to include human values alignment
Guidelines for ethical, safe, and responsible use of ChatGPT in educational settings
Investigation of human-chatbot relationships and their impact on student learning outcomes
Standards and regulatory frameworks for treating AI systems as co-authors or agents in academic contexts

Data: Not available. Paper states 'Availability of data and materials: Not applicable'Code: None mentionedExtracted from: pdfAgreement 56%

Explore related topics

AI Literacy Training Doctoral Training Case Study