AI evidence extraction

Evaluating AI-powered learning assistants in engineering higher education with implications for student engagement, ethics, and policy

Ramteja Sajja, Yusuf Sermet, Brian Fodale, İbrahim Demir · Scientific Reports · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

6/10

Relevance

3/4

Quality (LMQS)

Evidence

Citations

119.06

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1038/s41598-026-39237-5

Methodology & findings

Study design

Mixed-methods explanatory sequential design combining pre- and post-usage surveys (Likert-scale and open-ended items), system-generated usage logs, and qualitative analysis of student-generated AI queries.

Sample

N = 65, 12 groups

Primary method

Repeated-measures pre-post design using both independent and paired comparisons. Descriptive and correlational analyses. Internal consistency reliability assessed using Cronbach's alpha. Inter-rater reliability evaluated using Cohen's Kappa. Post hoc power analysis conducted assuming medium effect size (d=0.5). Quantitative analysis used to examine changes in student perceptions; qualitative analysis using hybrid coding process (automated LLM-based classifier followed by manual human review).

Main result

The study found that "students valued the AI assistant for its accessibility and comfort, with nearly half reporting greater ease using it than seeking help from instructors or teaching assistants." Additionally, "the tool was most helpful for completing homework and understanding concepts, though views on its instructional quality were mixed." A critical finding was that "ethical uncertainty, particularly around institutional policy and academic integrity, emerged as a key barrier to full engagement."

Reports effect sizes and confidence intervals.

Research paradigm

mixed-methods (quantitative and qualitative empiricism)

Author conclusions

"This study highlights the complex, dual nature of student engagement with domain-specific AI assistants in engineering education. While students embraced the Educational AI Hub for its accessibility and pragmatic utility in 'unblocking' specific academic tasks, their engagement was tempered by significant ethical anxiety and a lack of institutional clarity. The results suggest that students do not view AI as a replacement for human instruction, but rather as a convenient supplement that, while not always superior in quality, offers a non-judgmental space for procedural help." The authors further conclude that "successful adoption of discipline-specific tools is not solely a matter of functionality or technical design; it depends heavily on how well the tool aligns with students' learning needs within an ethically supported environment."

Risk of bias

Selection bias: Voluntary participation with self-selection into survey completion; Social desirability bias: Self-reported data on sensitive topics (academic integrity, trust); Novelty effect: Initial engagement may reflect curiosity rather than sustained adoption; Attrition: 18 students completed only pre-survey, 3 completed only post-survey; Institutional bias: Single university context limits generalizability; Extra credit incentive: May inflate participation and bias responses; Social desirability bias in self-reported survey data on academic integrity and trust; Novelty effect - student engagement may reflect initial curiosity rather than sustained adoption; Selection bias - 84% participation rate but voluntary participation with extra credit incentive; Single-institution sample limits generalizability; Unequal course enrollment (60 vs. 17 students) necessitated data aggregation; Selection bias: Voluntary participation with extra credit incentive (65 of 77 students enrolled; 84% participation rate); Social desirability bias: Self-reported survey data on sensitive topics (academic integrity, trust); Novelty effect: Engagement patterns may reflect initial curiosity rather than long-term adoption habits despite multi-phase introduction; Attrition: Unequal survey completion (18 pre-only, 44 both, 3 post-only); Generalizability: Single institution, engineering disciplines only; limited demographic diversity for generalizing across disciplines

Limitations

"First, the sample (N=65) was drawn from a single university and specific engineering disciplines, which limits the direct generalizability of findings to humanities or social science contexts where AI usage patterns may differ
Second, the reliance on self-reported survey data introduces potential social desirability bias, particularly regarding sensitive topics like academic integrity and trust
Finally, although we implemented a multi-phase introduction to mitigate the 'novelty effect,' student engagement patterns may still reflect initial curiosity rather than long-term adoption habits."

Open questions raised

Future research should employ multi-institutional samples and longitudinal designs to validate patterns over extended academic periods. More research is needed on how AI tools impact long-term learning behaviors, self-regulation, and academic outcomes across diverse disciplines and student populations. Additional work is required to develop ethical AI literacy frameworks that help students critically evaluate and appropriately use AI technologies in their academic work.
Impact of AI tools on long-term learning behaviors, self-regulation, and academic outcomes across diverse disciplines
Development of ethical AI literacy frameworks to help students critically evaluate and appropriately use AI technologies
Multi-institutional studies to validate engagement patterns over extended academic periods
Longitudinal designs tracking sustained adoption habits beyond initial novelty effect
Multi-institutional samples needed to validate patterns across diverse contexts

Data: The paper states: "The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request."; The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.; "The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request."Extracted from: pdfAgreement 53%

Explore related topics

Responsible AI Use Human-AI Collaboration Mixed Methods