AI evidence extraction

Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance

Yizhou Fan, Luzhen Tang, Huixiao Le, Kejie Shen, Shufang Tan, Yüan Shen et al. · British Journal of Educational Technology · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

6/10

Relevance

2/4

Quality (LMQS)

Evidence

419

Citations

147.12

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1111/bjet.13544

Methodology & findings

Study design

Randomized controlled experiment (RCT) with four experimental conditions conducted in a lab setting.

Sample

N = 117, 6 groups

Primary method

ANOVA followed by Tukey's HSD post-hoc test for intrinsic motivation comparisons; Kruskal-Wallis test followed by Mann-Whitney U test for non-parametric comparisons of self-regulated learning process frequency; process mining using first-order Markov Model (FOMM) implemented in pMineR software for temporal sequence analysis; overlay process mining for visualization of process models across groups

Main result

The study found that "learners who received different learning support showed no difference in post-task intrinsic motivation" and "ChatGPT group outperformed in the essay score improvement but their knowledge gain and transfer were not significantly different." Additionally, "there were significant differences in the frequency and sequences of the self-regulated learning processes among groups," and importantly, "AI technologies such as ChatGPT may promote learners' dependence on technology and potentially trigger 'metacognitive laziness'."

Reports effect sizes and confidence intervals.

Research paradigm

Positivist/empiricist - quantitative experimental design with statistical hypothesis testing

Author conclusions

The authors conclude: "In conclusion, our study highlights the potential of ChatGPT in improving essay scores, significantly outperforming other groups, including those guided by human experts. However, there were no significant differences in knowledge gain or transfer, indicating that while ChatGPT can enhance short-term task performance, it may not boost intrinsic motivation or long-term learning outcomes. The study also raises concerns about metacognitive laziness, where learners become overly reliant on AI, potentially hindering their ability to self-regulate and engage deeply in learning. This study contributes to the field of hybrid intelligence by revealing the potential and issues of learning with GenAI, and it calls for future research to deepen our understanding of how learners learn, regulate, collaborate and evolve with AI."

Risk of bias

Gender imbalance (70% female) limiting representativeness; Single-task design may not generalise to other learning contexts; Lab setting artificial conditions may not reflect real-world learning; Potential experimenter demand characteristics with awareness of ChatGPT restrictions; Self-selection bias in university student recruitment; Selection bias: Participants were primarily from a single university (Peking University) and English as second language speakers with Chinese as first language, limiting generalizability; Gender bias: 70% female participants, creating gender imbalance that may affect external validity; Hawthorne effect: Lab setting may alter participant behavior due to awareness of being observed; Task specificity bias: Single writing task may not generalize to other learning domains or task types; Attrition risk: Not explicitly discussed; unclear if all 117 participants completed all phases; Instructor/experimenter bias: Human expert condition involved subjective interactions that may not be standardized; Measurement bias: ChatGPT group may have discovered ways to circumvent task restrictions ('some learners would subsequently copy and paste content generated by ChatGPT'); Lab setting may not reflect real-world learning conditions (ecological validity concern); Single task type (writing task) limits generalizability; Potential expectancy effects from group assignment; Limited information on randomization concealment procedures provided in abstract; Lab setting may reduce ecological validity and generalisability to real-world learning contexts; Single writing task may not be representative of diverse learning scenarios; Potential selection bias in recruitment of 117 university students (convenience sample characteristics not detailed); Attrition or dropout rates not reported; Potential experimenter effects or demand characteristics in lab-based study

Limitations

The authors acknowledge several limitations: "The observed lack of significant differences between groups could be attributed to constraints related to task duration and sample size
Therefore, further research with larger sample sizes and exploration of long-term effects on motivation and performance is necessary." Additionally, "The study recruited 117 university students, of whom 70% were female
This gender imbalance may also limit the representativeness of the sample." They also note "Another limitation lies in the study's reliance on a single task involving reading and writing activities
Focusing exclusively on this task may not capture the diversity of cognitive and metacognitive processes engaged in various learning activities." Finally, "The last limitation is the lack of targeted and matured measures for assessing metacognitive laziness within the study's design."

Open questions raised

Lack of comparative research on learners engaging with different agents (AI, human experts, learning tools, no support)
Limited understanding of mechanisms and outcomes of hybrid human-AI learning based on empirical research
Need for research on how learners develop metacognitive skills while collaborating with AI
Necessity for multi-task and cross-context studies to test ChatGPT effectiveness
Gap in understanding long-term effects on motivation and performance
Need for targeted measurement protocols to assess metacognitive laziness

Data: "The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions."Extracted from: pdfAgreement 51%

Explore related topics

Productivity Effects Human-AI Collaboration Experimental