AI evidence extraction

Relying on LLMs: Student Practices and Instructor Norms are Changing in Computer Science Education

Xinrui Lin, Heyan Huang, Shumin Shi, John Vines · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Qualitative user studies with 16 CS students and 6 instructors.

Sample

N = 22, 6 groups

Primary method

Qualitative thematic analysis with three coding rounds: (1) scenario coding, (2) intent coding within scenarios, (3) thematic coding for RQ-related statements. No quantitative statistical tests performed. Analysis conducted independently by first author with no inter-rater reliability checks reported.

Main result

The study found that students have developed increasingly complex workflows for LLM use across five scenarios in CS education. Key findings reveal "varying levels of conflict between student practices and instructor norms, ranging from clear conflict in 'Writing-generation' and '(Programming) quiz-solving', through partial conflict in 'Programming project-implementation' and 'Project-based learning', to broad agreement in 'Writing-revision & ideation', '(Programming) quiz-correction' and 'Info-query & summary'." Additionally, "instructors are shifting from prohibiting to recognizing students' use of LLMs for high-quality work, integrating usage records into assessment grading."

Reports effect sizes.

Research paradigm

Qualitative empirical research (user studies with thematic analysis)

Author conclusions

"In this paper, we provide a nuanced, intent-based analysis of LLM adoption in Computer Science education, uncovering the friction between student pragmatism and instructor pedagogical goals. Our findings highlight that while broad agreement exists for using LLMs as reflective partners and information retrieval tools, conflicts persist in tasks for which students can directly obtain answers from LLMs. To adapt to students' increasingly sophisticated LLM utilization strategies, instructor norms are shifting from simple prohibitions toward recognizing students' use of LLMs to achieve high-quality outcomes, while incorporating such usage records into assessment. Furthermore, instructors propose that LLM design must adapt to current student practices: implementing default guardrails with game-like or empathetic elements to prevent students from 'deserting' to answer-giving models in high-conflict intents, particularly in 'Writing-generation', while integrating comprehension checks in low-conflict intents."

Risk of bias

Selection bias: Convenience sampling of CS students and instructors via university email and social media; Small sample size (16 students, 6 instructors) limiting generalizability; Geographic limitation: All participants from China, may not generalize to other regions; Measurement bias: Instructor norms derived after reviewing student practices may introduce bias in instructor assessment; Potential social desirability bias: Students and instructors may modify responses based on perceived expectations; Self-selection bias: Participants volunteered and had mean LLM experience of 3.5/5, potentially skewing toward experienced users; Selection bias: Participants recruited via university email and social media (self-selection); Social desirability bias: Students may underreport or misrepresent copying behaviors in interviews; Researcher bias: First author conducted all interviews and coding, no inter-rater reliability reported; Order bias: Instructor norms derived after reviewing student practices, introducing potential confirmatory bias; Geographic limitation: All participants from China, limiting generalizability; Sample homogeneity: All participants had prior LLM experience and relatively high LLM proficiency (mean=3.5/5); Selection bias: Participants self-selected via email and social media recruitment; may not represent all CS students/instructors; Sampling bias: All participants from China; CS pedagogy may vary across regions; Interviewer bias: First author conducted all sessions; potential for leading questions; Sequence bias: Instructor norms were elicited after reviewing student practices, potentially introducing bias in instructor responses; Small sample size: 16 students and 6 instructors limits generalizability; Attrition/missing data: No mention of attrition rates or missing data handling

Limitations

"This research is fundamentally qualitative
Student ways of handling LLM outputs and their views, as well as instructor norms, do not represent all CS students and instructors in higher education
Specifically, our collected intents are not exhaustive
for example, we did not capture writing intents in which students use LLMs to evaluate their own drafts, a practice that is common prior to submitting a paper
Moreover, CS pedagogy may vary across regions
Our students and instructors are from China, so while this study offers theoretical grounding for Chinese CS education, certain details in other regions require further verification

Open questions raised

Large-scale surveys needed to comprehensively document LLM use scenarios and intents in CS education and synthesize robust pedagogical innovations and LLM design recommendations
Need for exploration of LLMs' impacts on disciplines beyond CS in higher education
Need for development of pedagogical strategies addressing partial conflicts in Project-based learning intent
Missing research on writing intents where students use LLMs to evaluate their own drafts
Need for cross-regional verification of findings beyond China
Large-scale surveys of students and instructors to comprehensively document LLM use scenarios and intents in CS education

Extracted from: pdfAgreement 62%

Explore related topics

AI Literacy Training Doctoral Training Mixed Methods