AI evidence extraction

Student Perceptions of ChatGPT Use in a College Essay Assignment: Implications for Learning, Grading, and Trust in Artificial Intelligence

Chad C. Tossell, Nathan L. Tenhundfeld, Ali Momen, Katrina Cooley, Ewart J. de Visser · IEEE Transactions on Learning Technologies · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

3/4

Quality (LMQS)

Evidence

185

Citations

18.75

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1109/tlt.2024.3355015

Methodology & findings

Study design

Mixed-methods pre-post study design with 24 undergraduate engineering participants.

Sample

N = 24, 3 groups

Primary method

Mixed-methods approach combining quantitative and qualitative analysis. Quantitative: Parametric tests (paired t-tests), non-parametric tests (chi-squared), and Bayesian hypothesis testing (Bayes factors BF10 interpreted using established thresholds: <3 anecdotal, 3-10 moderate, 10-30 strong, 30-100 very strong, >100 extreme). Cohen's d used for effect sizes (small d=.2, medium d=.5, large d=.8). Linear regression model predicting intent to rely on ChatGPT from trust propensity and perceived trustworthiness. Qualitative: Thematic analysis using ChatGPT (v4.0, May 24, 2023) for coding with human author verification. Software: Not explicitly named; Bayes factors computed (likely JASP or similar).

Main result

The study found that "ChatGPT did not simplify the writing process. Instead, the tool transformed the student learning experience yielding mixed responses." Participants reported finding ChatGPT valuable for learning, and "their comfort with its ethical and benevolent aspects increased post-use." Additionally, "Students preferred instructors to use ChatGPT to help grade their assignments, with appropriate oversight. They did not trust ChatGPT to grade by itself." The thematic analysis revealed "a shift in student perception, evolving from viewing ChatGPT as a potential 'cheating tool' to recognizing it as a collaborative resource requiring human oversight."

Reports effect sizes and confidence intervals.

Research paradigm

Pragmatist (mixed-methods)

Author conclusions

"Our research indicates that while AI tools like ChatGPT have promising applications in higher education, they also pose challenges. These tools should be viewed as helpful assistants to enhance writing, learning, and grading and not as replacements for student effort or teaching oversight in grading. Effective and ethical use of AI in education requires acknowledging its limitations, fostering AI literacy, and developing proper assessment methods." The authors further conclude that "technologies like ChatGPT do not eliminate the need for student and instructor engagement, but rather complement it, requiring judicious trust and a blend of human skill and AI capabilities."

Risk of bias

Small sample size (n=24) may limit generalizability and statistical power to detect smaller effects; Self-selection bias (participation voluntary; 24 of 47 enrolled completed both surveys); Selection bias: Participants from single engineering course at USAF Academy; may not represent broader student population; Potential experimenter bias in qualitative coding mitigated by ChatGPT-assisted analysis with human verification; Participant characteristics (senior engineering students with human factors training) may influence responses to survey methodology; Hawthorne effect: Students aware their experiences were being studied may have modified behavior; Temporal confound: Timing coincided with ChatGPT's novelty (Spring 2023), limiting generalizability to current contexts where ChatGPT is ubiquitous; Selection bias: Small, non-random sample (24 of 47 enrolled participants); Voluntary participation bias: Participation was voluntary and ungraded; Specialized population: USAF Academy engineering students with human factors training, potentially more attuned to survey methodology; Attrition risk mitigated: All 24 pre-survey completers also completed post-survey; Self-report bias: Reliance on self-reported learning value and perceptions; Demand characteristics: Participants may have perceived expectations about AI adoption; Experimenter bias in qualitative coding: Though mitigated by use of ChatGPT for coding and author verification; Small sample size (n=24) limiting generalizability; Selection bias: participants were USAF Academy cadets with specialized engineering training, not representative of general student population; Participant education bias: human factors training may have influenced responses to survey methodology; Attrition: 24 of 47 enrolled completed both pre and post surveys; Structured assignment constraint may not reflect real-world ChatGPT use patterns; Instructor bias: same instructor graded all essays and may have influenced student perceptions

Limitations

"Most notably, the sample size was relatively small taking advantage of timing where ChatGPT had not been widely used by students yet and certainly not incorporated into curriculums
Low sample sizes are not uncommon with early studies on technology integration including for ChatGPT, smartphone use and robots presenting a trade-off between the impact of novel technology use and the generalizability of results." Additionally, "the assignment was highly structured and specified how students had to work with ChatGPT in three iterative steps to show contributions of the AI versus the student
However, this may have constrained the use of ChatGPT in other more creative ways or in ways that suited the student better." The authors also note that "the participants in this study were also senior-level undergraduate human systems engineering students
Their education in human factors processes including knowledge elicitation through survey-based user feedback methods (the approach used in this study) could have influenced their responses."

Open questions raised

Long-term investigations tracking students' evolving experiences and perceptions of AI-powered tools over extended periods
Larger sample sizes needed to identify smaller effects and enhance generalizability beyond engineering students
Research on where ChatGPT can be most effective in collaborative writing processes versus independent operation
Studies examining different assignment structures and degrees of constraint on ChatGPT use
Investigation of factors influencing trust and confidence in AI-generated outputs in high-stakes grading contexts
Research on developing AI literacy and proper assessment methods for AI-augmented learning

Data: Authors state "Please contact the authors for access to the assignment and/or rubric used in this study." No public dataset repository or DOI provided.; None explicitly mentioned as publicly available. Authors state: "Please contact the authors for access to the assignment and/or rubric used in this study."; Not explicitly stated. Authors note: "Please contact the authors for access to the assignment and/or rubric used in this study."Code: None mentionedExtracted from: pdfAgreement 51%

Explore related topics

Responsible AI Use Human-AI Collaboration Mixed Methods