AI evidence extraction

A Learning Agreement for Generative AI Use in University Courses: A Pilot Study

Marc Beardsley, Patrícia Santos, Ishari Amarasinghe, Emily Theophilou, Milica Vujović, Davinia Hernández‐Leo · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.36227/techrxiv.171078030.08340862/v2

Methodology & findings

Study design

Pilot study with pre-post surveys, thematic analysis of student artefacts (group assignments), and qualitative analysis of student suggestions for improvement

Sample

N = 17, 2 groups

Primary method

Thematic analysis of student suggestions and artefacts; descriptive analysis of survey responses. Specific statistical software and formal hypothesis testing procedures are not mentioned in the abstract.

Main result

Results show that "the vast majority of students were in favour of the learning agreement approach both at the start and upon completion of the course." However, "7 of 17 groups did not use GenAI in their assignment. Of the 10 groups that did, only 1 acknowledged GenAI limitations in adherence with the learning agreement." This suggests a significant gap between student agreement with ethical principles and actual implementation in practice.

Reports effect sizes.

Research paradigm

Mixed methods (qualitative and quantitative descriptive)

Author conclusions

The authors conclude that "learning agreements have the potential to offer an interface through which student decision-making can be supported and interactions among students, educators, researchers, and policy makers related to the ethical and societal challenges of GenAI can take place." This suggests a framework-level contribution despite limited adherence to specific ethical principles in practice.

Risk of bias

Small sample size (17 groups, ~likely fewer than 100 students); Single course/institution setting - limited generalizability; Self-selection bias - students willing to participate in learning agreement study; Social desirability bias in survey responses; No control group for comparison; Attrition risk not reported; Selection bias: Single institution, single course, self-selected student sample; Small sample size (17 groups) limits generalizability; Social desirability bias: Students may overstate agreement with learning agreement in pre-post surveys; Attrition not reported; No inter-rater reliability reported for thematic analysis; No blinding mentioned for artefact analysis; Selection bias: Self-selected student population in a first-year engineering course; Small sample size: Only 17 groups limits generalizability; Potential social desirability bias: Students may have reported positive views of the learning agreement without genuine commitment to its principles; Lack of control condition: No comparison group without learning agreement

Open questions raised

Need for larger-scale studies beyond pilot phase
Implementation of suggested improvements: making agreements easier to understand with specific examples
Re-engagement with agreement during the course
Better strategies for encouraging GenAI acknowledgment of limitations
Broader stakeholder engagement (educators, researchers, policymakers)
Need for longitudinal studies to assess sustained impact of learning agreements

Data: not_statedCode: not_statedExtracted from: pdfAgreement 60%

Explore related topics

Institutional Policy AI Governance Case Study