AI evidence extraction

Auditing Student-AI Collaboration: A Case Study of Online Graduate CS Students

Nifu Dan · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

8/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed-methods audit using two sequential surveys.

Sample

N = 44, 8 groups

Primary method

Descriptive statistics (means of Likert-scale responses). Aggregate desire level and aggregate usage level computed as means across participants for each task. Four-zone classification framework applied to map tasks into quadrants based on desire-usage alignment. Qualitative thematic analysis of open-ended responses using iterative coding to identify recurring design themes and system feature expectations. No inferential statistical tests, hypothesis testing, or confidence interval calculations reported.

Main result

The study found that "students' adoption of AI is highly task dependent, with consistent gaps between desired and actual automation that reflect cautious, evaluative decision-making rather than indiscriminate use." Across most tasks, "desired automation levels exceed reported usage, indicating a consistent gap between participants' preferences for AI involvement and their current patterns of AI use." Tasks that are "procedural, supportive, or easily verifiable (e.g., writing revision, summarization, citation formatting, and debugging code) tend to cluster in regions of both high desired automation and high usage," while tasks that are "socially sensitive (e.g., drafting emails) or cognitively generative (e.g., brainstorming, quantitative reasoning) exhibit larger gaps."

Reports effect sizes.

Research paradigm

Mixed-methods empirical research combining quantitative survey design with qualitative analysis

Author conclusions

The authors conclude: "Through a mixed-methods user study, we showed that students' adoption of AI is highly task dependent, with consistent gaps between desired and actual automation that reflect cautious, evaluative decision-making rather than indiscriminate use. We further demonstrated that efficiency-driven motivations coexist with task-specific concerns, particularly around inaccurate information and reduced critical thinking. Finally, we identified key design expectations for educational AI systems, including transparency, verifiability, uncertainty communication, and support for user skepticism. Together, our findings provide empirical grounding and design-relevant insights for building educational AI systems that align with students' learning goals and preserve human agency in student–AI collaboration."

Risk of bias

Selection bias: Participants were self-selected volunteers from a single online graduate program (OMSCS); Selection bias: Only students reporting at least occasional AI use were retained (N=44 from N=57), excluding non-users; Sample bias: Homogeneous population (graduate CS students), limiting generalizability; Recall bias: Self-reported actual AI usage rather than objective measures; Desirability bias: Responses about automation preferences and concerns may reflect socially desirable answers; Measurement validity: Likert-scale responses to 'actual usage' conflate perceived capability with habitual use rather than objective system performance; Selection bias: Voluntary participation with filtering for AI users, excluding non-users from analysis (N=57 initial, N=44 final); Self-report bias: Actual AI usage measured through self-reported frequency and reliance rather than objective system logs; Sample homogeneity: Limited to graduate computer science students at one institution (Georgia Tech OMSCS program); Specialization bias: Participants heavily represented AI, Machine Learning, and HCI specializations, not representative of broader CS student population; Selection bias: Participants were filtered to only include those reporting at least occasional AI use (N=44 retained from N=57), excluding non-users and rare users; Self-report bias: Actual AI usage was operationalized through self-reported frequency rather than objective measurement; Population specificity: Sample drawn exclusively from online CS graduate students at one institution (Georgia Tech OMSCS), limiting generalizability; Lack of random assignment: No control condition or comparison group structure

Limitations

The authors acknowledge that "in future work, this study can be extended to include a broader range of academic tasks and a more diverse student population beyond computer science students." Additionally, they note it "would be beneficial to design the survey with open-ended questions for capturing students' concerns and reasons for using AI" in the initial phase, as "open-ended responses would allow participants to articulate more nuanced, context-dependent perspectives that may not fit within predefined categories."

Open questions raised

Authors identify the following future directions: (1) extend study to include "a broader range of academic tasks and a more diverse student population beyond computer science students"; (2) expand survey design with more open-ended questions to capture "more nuanced, context-dependent perspectives that may not fit within predefined categories."
The authors identify the following gaps and future directions: (1) Extend the study to include a broader range of academic tasks beyond the 12 examined; (2) Include a more diverse student population beyond computer science students; (3) Design surveys with open-ended questions for capturing students' concerns and reasons for using AI from the outset, rather than as a follow-up phase, to allow participants to articulate more nuanced, context-dependent perspectives that may not fit within predefined categories.
Need to extend the study to include broader range of academic tasks beyond the 12 examined
Need for more diverse student population beyond computer science students
Opportunity to use open-ended questions for capturing concerns and reasons for using AI in initial survey phase to capture more nuanced, context-dependent perspectives
Gap in understanding student-AI collaboration preferences across diverse educational tasks and populations

Extracted from: pdfAgreement 58%

Explore related topics

Responsible AI Use Human-AI Collaboration Mixed Methods