AI evidence extraction

Influence of Prompts Structure on the Perception and Enhancement of Learning through LLMs in Online Educational Contexts

Silvia Rodriguez-Donaire · IntechOpen eBooks · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

11.48

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.5772/intechopen.1006481

Methodology & findings

Study design

Mixed-methods empirical study combining quantitative analysis (ANOVA, GLM, Random Forest models) and qualitative analysis (coding of student responses, classroom observations).

Sample

N = 182, 8 groups

Primary method

ANOVA (Analysis of Variance) with interaction effects; Generalized Linear Model (GLM) with logistic regression; Random Forest classification model; ROC curve analysis and AUC calculation; Descriptive statistics (mean, median, SD, SE, kurtosis, skewness); Manual coding by authors with AI (ChatGPT 4.0) assistance, followed by manual verification and correction

Main result

The study found that "well-structured prompts significantly improve students' perception of the depth and accuracy of GenAI-generated responses, leading to a more effective learning process." The results specifically demonstrate that "A well-designed prompt structure contributes positively to the perception of depth and accuracy in LLM responses" and "The positive perception of the depth and accuracy of AI responses significantly improves the student's learning process."

Reports effect sizes.

Research paradigm

Mixed-methods (quantitative and qualitative)

Author conclusions

"The study examined how the structure of prompts used in LLM models affects the perception of depth, accuracy, and effectiveness of the student's learning process. The results, obtained through both quantitative and qualitative analyses, confirm several crucial hypotheses regarding the relationship between prompt structure and the perception and effectiveness of AI responses in the learning process." The authors conclude that "By better understanding these dynamics, we can develop strategies to improve prompt design in educational contexts. This will allow us to obtain better responses generated by LLM, ensuring that both prompt effectiveness and students' positive perception of learning are maximized."

Risk of bias

Selection bias: Only 182 of 304 enrolled students completed the activity and met inclusion criteria; 12 students excluded due to missing prompt data; Lack of randomization: No control group; all students received the intervention; Single context: Study limited to one online university course (Business Administration at UOC); Subjectivity in coding: AI-assisted coding required manual corrections by authors, introducing potential coder bias; Collinearity: VERACITY, OBJECTIVES, CONTEXT, and SPECIFIC variables eliminated due to lack of variability and collinearity, limiting hypothesis testing; Selection bias: Only 182 of 304 enrolled students participated and had complete data (60% attrition); manual data entry and coding corrections by authors introduce subjective bias; Measurement bias: Coding performed by ChatGPT 4.0 with manual author corrections; some cases had 'not completely correct' coding; Context-specific bias: Single institution (UOC), single course subject (Information Systems), asynchronous online format limits generalizability; Confounding: Classroom differences showed highly significant effects (p<0.001) but not thoroughly investigated; Collinearity: VERACITY, OBJECTIVES, CONTEXT, and SPECIFIC variables removed from analysis due to lack of variability and collinearity, limiting theoretical testing; Selection bias: Only 182 of 304 students fully completed the study (60% participation rate); Attrition: 12 students excluded due to missing prompt data; Classroom effects: Highly significant classroom variability (p < 0.001) suggests potential instructor or contextual confounding; Manual coding bias: Although AI-assisted coding was used, authors manually modified some codes, introducing potential subjectivity; Collinearity: VERACITY and other key variables eliminated from ANOVA model due to collinearity, limiting analysis; Lack of control group: No randomized comparison condition

Limitations

The authors acknowledge that "there was a lack of variability in responses regarding the efficient structure of the prompt and truthfulness of the LLM response, which limited the quantitative analysis and made the results dependent on qualitative data and subjective perceptions." Additionally, "the specific use of LLMs for a particular activity means that the results cannot be generalized to all LLM application scenarios or educational disciplines" and "although the model was good, a larger sample size could help produce more reliable results."

Open questions raised

Need for larger and more diverse sample sizes to encompass broader range of variability in perceived truthfulness and other evaluated metrics
Investigation of whether type of generative AI used impacts confirmation of hypotheses
Experimental studies with systematic manipulation of prompt characteristics to directly evaluate their impact on quality and perception of AI responses
Extension to different educational disciplines and types of questions to explore applicability and effectiveness in various educational contexts
Development of analytical tools enabling teachers and developers to evaluate and optimize prompt structure in real time
Expand the database by including a larger and more diverse sample of responses to encompass broader variability in perceived truthfulness and other evaluated metrics

Data: Engineering prompts for codifying students' prompt structure and understanding their learning perception from receiving feedback on an online activity using AI; Open repository containing coding instructions and experimental data; reference [26]: Rodriguez Donaire S. Engineering prompts for codifying students' prompt structure and understanding their learning perception from receiving feedback on an online activity using AI. CORA, Research Data Repository. 2024;V1:UNF:6:VIaUGgwuP9lPi3I4YBwLZw== [fileUNF]. DOI: 10.34810/data1554; Rodriguez Donaire S. Engineering prompts for codifying students' prompt structure and understanding their learning perception from receiving feedback on an online activity using AI. CORA, Research Data Repository. 2024;V1:UNF:6:VIaUGgwuP9lPi3I4YBwLZw== [fileUNF]. DOI: 10.34810/data1554Extracted from: pdfAgreement 53%

Explore related topics

AI Manuscript Writing Academic Writing Experimental