AI evidence extraction

Cognitive ease at a cost: LLMs reduce mental effort but compromise depth in student scientific inquiry

Matthias Stadler, Maria Bannert, Michael Sailer · Computers in Human Behavior · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

8/10

Relevance

1/4

Quality (LMQS)

Evidence

147

Citations

96.35

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1016/j.chb.2024.108386

Methodology & findings

Study design

Randomized controlled trial (RCT) with two groups: students were randomly assigned to either use ChatGPT3.5 or Google to research a socio-scientific issue and provide recommendations and justifications..

Sample

N = 91, 5 groups

Main result

The study found that "students using LLMs experienced significantly lower cognitive load" while "these students demonstrated lower-quality reasoning and argumentation in their final recommendations compared to those who used traditional search engines." Additionally, "the homogeneity of the recommendations and justifications did not differ significantly between the two groups, suggesting that LLMs did not restrict the diversity of students' perspectives."

Reports effect sizes.

Research paradigm

Empirical-analytical (positivist with mixed-methods elements)

Author conclusions

The authors conclude that "while LLMs can decrease the cognitive burden associated with information gathering during a learning task, they may not promote deeper engagement with content necessary for high-quality learning per se." This highlights "the nuanced implications of digital tools on learning" and suggests a trade-off between cognitive ease and learning depth.

Risk of bias

Selection bias: students self-selected into study participation; Confounding variables: prior experience with LLMs or search engines not controlled; Generalizability: single task type (nanoparticles in sunscreen) may not generalize to other domains; Cognitive load measurement: self-reported measures prone to demand characteristics; Single institution sample (university students only); Limited to one specific socio-scientific task (nanoparticles in sunscreen); Potential confounding by prior familiarity with tools; No apparent control for individual differences in cognitive ability; Single time-point measurement (no longitudinal follow-up); Potential selection bias if self-selection into study participation; Potential confounding variables related to prior experience with LLMs vs search engines; Potential experimenter effects based on tool familiarity; Single task/topic (nanoparticles in sunscreen) limits generalizability

Limitations

The authors note limitations related to the specific task context and population, though a detailed limitations section is not explicitly provided in the abstract
The study was limited to "a total of 91 university students" in a specific learning context involving "the socio-scientific issue of nanoparticles in sunscreen."

Open questions raised

The study identifies the need for further research on the mechanisms underlying the trade-off between reduced cognitive load and lower reasoning quality when using LLMs, and suggests investigation of task types, domains, and student populations to understand generalizability.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 64%

Explore related topics

Productivity Effects Human-AI Collaboration Experimental