12,445 papers · continuously updated · last export: 10 Aug 2026livingmeta.ai
← Browse all papers
AI evidence extraction

Studying the GenAI’s Impact on Learning and Critical Thinking: A Multi-Site Field Experiment

Journal of the Association for Information Systems · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

8/10
Relevance
1/4
Quality (LMQS)
E
Evidence
0
Citations
0.00
FWCI

Methodology & findings

Study design

Multi-site randomized controlled field experiment with 270 students across ten universities and polytechnics, randomly assigned to experimental (ChatGPT) and control (traditional resources) groups.

Sample

N = 270, 2 groups

Main result

The study found that "ChatGPT enhanced content comprehension and motivation, but many students struggled with evaluating AI reliability and creating effective prompts." This suggests that while GenAI tools provide certain pedagogical benefits in terms of student engagement and understanding, critical limitations exist in students' ability to critically evaluate and effectively utilize these tools.

Reports effect sizes and confidence intervals.

Research paradigm

Positivist/Empiricist

Author conclusions

The authors conclude that "Generative AI (GenAI) tools are transforming higher education, but robust evidence of their pedagogical value is limited." They further note that while GenAI shows promise for enhancing comprehension and motivation, significant challenges remain regarding student critical evaluation of AI outputs and prompt engineering skills.

Risk of bias

Potential selection bias: Not specified how students were recruited across ten sites; Attrition risk: Not reported whether all 270 students completed the study; Confounding variables: Local learning outcomes varied across institutions, which may confound treatment effects; Hawthorne effect: Students aware of being observed during classroom observations may alter behavior; Technology access bias: Only students with access to licensed ChatGPT included in experimental group; Selection bias: Potential differences between students willing to use GenAI versus control group; Attrition bias: Not mentioned in abstract; unclear if follow-up was complete across all sites; Confounding variables: Site-level differences (university vs. polytechnic) may affect outcomes; Hawthorne effect: Students in experimental group may behave differently due to awareness of monitoring; Instructor variability: Different instructors across ten sites may implement interventions inconsistently; Selection bias potential across ten different institutions; Attrition risk with 270 students across multiple sites; Potential confounding from institutional differences in learning outcomes and teaching practices; Hawthorne effect from classroom observations

Open questions raised

  • The authors identify the limited robust evidence of GenAI's pedagogical value as a key gap. Future research directions include investigating how to better support students in evaluating AI reliability and developing effective prompting strategies.
  • The abstract indicates that "robust evidence of their pedagogical value is limited," suggesting a gap in empirical understanding of GenAI's educational impact that this study addresses.
Data: not_statedCode: not_statedExtracted from: pdfAgreement 62%

Explore related topics

AI Literacy TrainingDoctoral TrainingExperimental

Related papers

← Browse all papers