AI evidence extraction

Is it harmful or helpful? Examining the causes and consequences of generative AI usage among university students

Muhammad Abbas, Farooq Ahmed Jam, Tariq Iqbal Khan · International Journal of Educational Technology in Higher Education · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

6/10

Relevance

4/4

Quality (LMQS)

Evidence

372

Citations

164.07

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s41239-024-00444-7

Methodology & findings

Study design

Two-phase empirical study: Study 1 involved exploratory factor analysis (EFA) with principal component analysis to develop and validate an 8-item ChatGPT usage scale using 165 university students.

Sample

N = 494, 7 groups

Primary method

Partial Least Squares Structural Equation Modeling (PLS-SEM) using SmartPLS software with 5,000 bootstrap samples; Exploratory Factor Analysis (EFA) with principal component analysis and varimax rotation; tests for Bartlett's sphericity, Kaiser-Meyer-Olkin (KMO) sampling adequacy; discriminant validity via Fornell-Larcker criteria and Heterotrait-Monotrait (HTMT) ratios; variance inflation factor (VIF) for multicollinearity; Cronbach's alpha (CA) and composite reliability (CR) for internal consistency; average variance extracted (AVE) for convergent validity

Main result

The study found that "students who experienced high levels of academic workload and time pressure to accomplish their tasks reported higher use of ChatGPT." Furthermore, "those students who frequently used ChatGPT were more likely to engage in procrastination than those who rarely used ChatGPT" and "students who frequently used ChatGPT also reported memory impairment." Additionally, "students who frequently used ChatGPT for their academic tasks had poor CGPAs," demonstrating that excessive ChatGPT usage had negative effects on academic performance.

Reports effect sizes and confidence intervals.

Research paradigm

positivist_quantitative

Author conclusions

"The findings suggested that those students who experienced high levels of academic workload and time pressure to accomplish their tasks reported higher use of ChatGPT... Furthermore, our findings suggested that excessive use of ChatGPT can have harmful effects on students' personal and academic outcomes." The authors conclude that "educators should encourage students to actively engage in critical thinking and problem-solving by assigning activities, assignments, or projects that cannot be completed by ChatGPT. This can mitigate the adverse effects of ChatGPT on their learning journey and mental capabilities."

Risk of bias

Attrition bias: 840 initial responses in phase 1 reduced to 494 complete responses by phase 3 (41% attrition rate); Selection bias: Participants were self-selected university students with access to online surveys; Social desirability bias: Addressed through procedural remedies but self-reported measures remain vulnerable; Common method bias: Addressed through three-wave time-lagged design and confidentiality assurances; Attrition bias: sample size dropped from 900 initial contacts to 494 complete responses across three waves (45.1% retention); Selection bias: convenience sampling of university students in Pakistan; Self-report bias: participants self-reported CGPA and other measures; Common method bias: partially addressed through temporal separation and procedural remedies but acknowledged as a potential limitation; Geographic limitation: study conducted only in Pakistan, limiting generalizability; Attrition bias: Sample decreased from 900 (Phase 1) to 840 (Phase 2) to 675 (Phase 2) to 540 (Phase 3), with 494 complete responses retained after missing data removal (45% attrition); Selection bias: Convenience sampling from Pakistani universities; may not represent global student populations; Self-report bias: Subjective measures for procrastination, memory loss, and most predictors; Social desirability bias: Despite remedies, self-reported academic misconduct behavior may be underreported; Common method bias: Although time-lagged design was used, single-method reliance (survey) throughout

Limitations

"Although we used a time-lagged design, as compared to cross-sectional designs used by prior research, we could not completely rule out the possibility of reciprocal relationships
For example, it is also possible that ChatGPT usage may also help lessen the subsequent perceptions of workload." Additionally, the study was limited to university students in Pakistan, and did not examine personality factors or contextual moderators that may influence quality consciousness effects on ChatGPT usage.

Open questions raised

The authors identify several gaps: (1) future research should examine causal mechanisms using longitudinal designs to rule out reciprocal relationships; (2) personality factors such as trust propensity and Big Five traits should be examined in relation to ChatGPT usage; (3) the insignificant effect of quality consciousness on ChatGPT usage warrants further investigation with contextual moderators; (4) fear of punishment as a deterrent to ChatGPT misuse should be explored; (5) benefits of generative AI use and differential effects across fields of knowledge and gender should be compared; and (6) long-term effects on students' learning, cognitive skills, mental health, and overall health outcomes need investigation.
Limited empirical research on factors driving ChatGPT usage among university students
Lack of empirical evidence on beneficial or harmful effects of ChatGPT usage
Need for examination of personality factors (trust propensity, Big Five traits) in relation to ChatGPT usage
Further investigation needed on quality consciousness and ChatGPT usage relationships
Need for longitudinal designs to examine reciprocal relationships and causal mechanisms

Data: "The data associated with this research is available upon a reasonable request."; "The data associated with this research is available upon a reasonable request." No publicly accessible repository or permanent DOI provided.Code: None mentioned.Extracted from: pdfAgreement 51%

Explore related topics

Productivity Effects Doctoral Training Mixed Methods