AI evidence extraction

Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances

Margherita Bernabei, Silvia Colabianchi, Andrea Falegnami, Francesco Costantino · Computers and Education Artificial Intelligence · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

0/4

Quality (LMQS)

Evidence

150

Citations

5.31

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1016/j.caeai.2023.100172

Methodology & findings

Study design

In-depth case study combining conceptualization and empirical evidence analysis.

Sample

1 group

Primary method

Qualitative and quantitative analysis combining conceptualization and empirical evidence; questionnaire administration based on technology acceptance model constructs; essay quality assessment; detector testing evaluation

Main result

The study found that "The essay assessment showed good results" when engineering students composed essays using ChatGPT, but "Thirteen LLMs detectors were tested without achieving satisfactory results, suggesting to avoid LLMs ban." Additionally, students' perceptions of usefulness and acceptance of LLMs in learning were assessed through questionnaire responses based on technology acceptance models.

Reports effect sizes.

Research paradigm

Mixed methods (deductive/inductive)

Author conclusions

The authors conclude that "The results contribute to qualitative evidence by highlighting possible future research and educational practices." They emphasize that rather than banning LLMs, the findings suggest focusing on detection limitations and student perception, stating the approach should be to understand "how students perceive the usefulness and acceptance of LLMs in learning" rather than restrictive measures.

Risk of bias

Selection bias (engineering students only); potential social desirability bias in questionnaire responses; limited generalizability from case study design; Selection bias: participants were volunteer engineering students (mechanical and management) who self-selected into the study. No randomization or control group mentioned. Detection bias: results from 13 different LLM detectors may introduce detection algorithm variability. Questionnaire response bias: students may provide socially desirable responses about technology acceptance.; Selection bias: Single institution/program (mechanical and management engineering students only); Self-selection bias: Students who chose to participate in LLM essay composition study; Social desirability bias: Questionnaire responses may reflect students' perceived acceptability rather than true perception; Lack of randomization: No control group mentioned

Open questions raised

The abstract indicates "few studies examine students' use of LLMs as learning tools" and that there is "a growing need to explore their application in education." Future research directions include better LLM detection systems and refined educational practices for LLM integration.
The authors identify the need for: (1) improved LLM detection systems beyond the 13 tested; (2) further investigation of students' use of LLMs as learning tools; (3) educational practices and policies appropriate for LLM integration; (4) future research directions that build on the qualitative evidence presented.
Further research on LLM detection systems and their improvement
Exploration of educational practices that accommodate LLM use
Investigation of how to optimize student learning with LLM assistance
Need for more studies examining students' use of LLMs as learning tools

Data: not_statedCode: not_statedExtracted from: pdfAgreement 53%

Explore related topics

AI Manuscript Writing Academic Writing Case Study