AI evidence extraction

The role of generative AI chatbots in higher education: A student-centric conceptual analysis of benefits, ethics, and privacy concerns

Alex Vallejo, Rubén Nicolás-Sans · Journal of Technology and Science Education · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

8/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.3926/jotse.3643

Methodology & findings

Study design

Mixed-methods pilot study combining semi-structured interviews, perception surveys, and controlled academic task evaluation with quantitative performance metrics

Sample

N = 420, 4 groups

Primary method

Descriptive statistics, ANOVA tests, and thematic coding of interviews. Quantitative performance metrics assessed included perplexity, response latency, context window, and SWE-bench accuracy.

Main result

The study found that "ChatGPT and Claude achieved the best balance between pedagogical clarity and privacy compliance, while Gemini excelled in technical capacity but showed weaker ethical safeguards." Additionally, "student satisfaction was strongly associated with transparency in data policies and pedagogical usefulness rather than raw technical performance."

Reports effect sizes.

Research paradigm

Mixed methods (qualitative and quantitative)

Author conclusions

The authors conclude that "These findings highlight the need for universities to adopt generative AI technologies under robust ethical and privacy frameworks." They emphasize that institutional adoption should be guided by consideration of both technical capabilities and ethical guardrails, with future research needed to extend findings through "longitudinal studies and institutional case analyses."

Risk of bias

Selection bias: Participants self-selected from five European universities only; Potential social desirability bias in interview responses regarding ethical concerns; Limited geographic scope (Europe only) may affect generalizability; Pilot study design with modest sample size; Selection bias: Self-selected participants (students and faculty who volunteered for interviews and surveys); Small sample for generalization: Pilot study design with 300 undergraduates and 120 faculty from only 5 European universities; Interviewer bias: Semi-structured interviews may be subject to interviewer effects; Lack of randomization: No mention of random assignment to conditions; Potential funding bias: Not disclosed in abstract; Selection bias: Volunteer participants from five European universities may not be representative; Social desirability bias: Survey responses regarding chatbot use; Lack of control group for comparison; Geographic limitation to European universities

Limitations

The abstract indicates this was "a pilot study" which inherently limits generalizability
The authors note that "Future research should extend this pilot into longitudinal studies and institutional case analyses," suggesting awareness of temporal and scope limitations, though explicit limitations are not detailed in the abstract provided.

Open questions raised

Future research should extend this pilot into longitudinal studies with longer-term tracking of student outcomes, institutional case analyses examining real-world implementation across different university contexts, and investigation of how ethical frameworks evolve as AI chatbot capabilities advance
The authors identify the need to "extend this pilot into longitudinal studies and institutional case analyses" as a critical future research direction, indicating gaps in understanding long-term impacts and real-world institutional adoption of these technologies.
"Future research should extend this pilot into longitudinal studies and institutional case analyses."

Data: not_statedCode: not_statedExtracted from: pdfAgreement 68%

Explore related topics

Responsible AI Use AI Governance Mixed Methods