AI evidence extraction

Using ChatGPT is easy, using it effectively is tough? A mixed methods study on K-12 students’ perceptions, interaction patterns, and support for learning with generative AI chatbots

Maria Klar · Smart Learning Environments · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

5/10

Relevance

3/4

Quality (LMQS)

Evidence

Citations

42.93

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s40561-025-00385-2

Methodology & findings

Study design

Mixed-methods experimental design combining quantitative survey data (cognitive load, knowledge gains, satisfaction) with qualitative analysis of chatbot interaction protocols.

Sample

N = 106, 3 groups

Primary method

One-way MANOVA (Multivariate Analysis of Variance) to test group differences across three dependent variables (adaptation prompts, extraneous load, knowledge gain). Post-hoc univariate ANOVAs conducted for each dependent variable. Moderation analyses performed using PROCESS macro by Hayes (2017). Interrater reliability assessed using weighted kappa (κ). Qualitative coding scheme developed deductively and refined inductively for chatbot interactions. Descriptive statistics reported as means and standard deviations; Cronbach's alpha reported for internal consistency of scales.

Main result

The study found that "students who received adaptation guidance used a greater variety and a significantly higher number of adaptation prompts, asked more follow-up questions, and engaged in less off-topic conversation" compared to the control group. However, "the adaptation guidance led to a significantly higher number of adaptation prompts, while extraneous load and knowledge gain did not differ significantly." Notably, "a third of the students without adaptation guidance did not query the chatbot for any adaptation," indicating underutilization of core affordances despite high perceived ease of use.

Reports effect sizes.

Research paradigm

mixed_methods_pragmatism

Author conclusions

The authors conclude: "Students generally perceived the chatbots favorably and reported high levels of satisfaction and ease of use, regardless of whether they received adaptation guidance. This perception of the chatbot as 'easy', suggests an underinvestment of mental effort (Salomon, 1984; Stadler et al., 2024) and, combined with students' moderate prior experience, a potential overestimation of their ability to effectively use these tools for learning." Furthermore, "the results demonstrate that even brief instruction on adaptation prompts can significantly influence how students interact with chatbots. Students who received adaptation guidance used a greater variety and a significantly higher number of adaptation prompts, asked more follow-up questions, and engaged in less off-topic conversation." They note that "Using the full range of chatbot capabilities in self-regulated learning might still be 'tough', but this can be effectively alleviated by targeted support."

Risk of bias

Selection bias: Participants were recruited from five secondary schools in Germany and self-selected to participate; Small group randomization rather than individual-level randomization may introduce clustering effects; Demand characteristics from the instructional intervention suggesting adaptation prompts are beneficial; Limited ecological validity due to restriction to chatbot-only use without web search; Short intervention duration (20 minutes) may not reflect sustained real-world use patterns; Demand characteristics from instructional framing in experimental group; Selection bias: volunteers from five schools may not be representative; No blinding of participants or facilitators to condition; Ecological validity limitation: restricted to chatbot-only interface without web search; Small sample size (n=106) limits power to detect small effects; Possible social desirability bias in survey responses regarding satisfaction and competence; Demand characteristics: The instruction to experimental group about adaptation prompts may have created task demands not perceived by control group; Selection bias: Participants were volunteers from five secondary schools, potentially self-selected for interest in technology; Lack of blinding: Not mentioned whether participants or researchers were blinded to conditions; Small group randomization: Randomization occurred at group level (up to 7 students per session, average 4), not individual level, which may introduce selection bias within groups; Low variance in moderator variables: Prior domain knowledge and chatbot experience were low and showed little variance, limiting moderation analyses; Short intervention duration: Single 20-minute session may not reflect sustained learning effects; Artificial constraint: No web search access differs from real-world student use

Limitations

The authors identify several limitations: "First, there might have been unintended demand characteristics to the instructional pointers on adaptation prompts that undermine an effect on extraneous load." "Second, while the premise of the study is close to an authentic setting, i.e., conducting research on an unfamiliar topic with the help of a chatbot, there is a limitation to its ecological validity: Students did not have access to web search and were restricted to the chatbot." "Third, the sample size is not large enough to detect small effects." "Fourth, it is important to note the short-term nature of this study
The research focused on a single, 20-min interaction with a chatbot, which may not fully capture the potential long-term effects of adaptation guidance."

Open questions raised

Long-term effects of adaptation guidance on learning outcomes and self-regulated learning skills over extended periods
How students switch between chatbots and web search in natural settings
Targeted interventions for other aspects of self-regulated learning such as investing appropriate mental effort and choosing effective strategies for planning, performing, and evaluating
More naturalistic settings where students can combine chatbots with web search and other tools
Interview studies to gain nuanced understanding of learner perceptions and interaction patterns
Long-term effects of adaptation guidance beyond single 20-minute interaction

Data: Not mentionedCode: Not mentionedExtracted from: pdfAgreement 63%

Explore related topics

Productivity Effects Human-AI Collaboration Experimental