AI evidence extraction

Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances

Margherita Bernabei, Silvia Colabianchi, Andrea Falegnami, Francesco Costantino · Computers and Education Artificial Intelligence · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

Evidence

150

Citations

5.31

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1016/j.caeai.2023.100172

Methodology & findings

Study design

In-depth case study combining conceptualization and empirical evidence analysis.

Sample

1 group

Primary method

Qualitative and quantitative analysis combining conceptualization and empirical evidence; questionnaire administration based on technology acceptance model constructs; essay quality assessment; detector testing evaluation

Main result

The study found that "The essay assessment showed good results" when engineering students composed essays using ChatGPT, but "Thirteen LLMs detectors were tested without achieving satisfactory results, suggesting to avoid LLMs ban." Additionally, students' perceptions of usefulness and acceptance of LLMs in learning were assessed through questionnaire responses based on technology acceptance models.

Reports effect sizes.

Research paradigm

Mixed methods (deductive/inductive)

Author conclusions

The authors conclude that "The results contribute to qualitative evidence by highlighting possible future research and educational practices." They emphasize that rather than banning LLMs, the findings suggest focusing on detection limitations and student perception, stating the approach should be to understand "how students perceive the usefulness and acceptance of LLMs in learning" rather than restrictive measures.

Risk of bias

Selection bias (engineering students only); potential social desirability bias in questionnaire responses; limited generalizability from case study design; Lack of randomization: No control group mentioned

Open questions raised

The abstract indicates "few studies examine students' use of LLMs as learning tools" and that there is "a growing need to explore their application in education." Future research directions include better LLM detection systems and refined educational practices for LLM integration.
The authors identify the need for:
improved LLM detection systems beyond the 13 tested
future research directions that build on the qualitative evidence presented.
Investigation of how to optimize student learning with LLM assistance

Data: not_statedCode: not_statedExtracted from: pdf

Explore related topics

AI Manuscript Writing Academic Writing Case Study