AI evidence extraction

Exploring Students’ Competence Development With Generative AI: A Design Science Research On Master’S Thesis Writing

Aurélie Dudézert, Olfa Chourabi, Thierno Tounkara · Journal of the Association for Information Systems · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Design Science Research Methodology (DSRM) with comparative cohort analysis.

Primary method

Design Science Research Methodology (Geerts, 2011) with experiential learning approach based on Kolb's four-stage cycle (Concrete Experience, Reflective Observation, Abstract Conceptualization, Active Experimentation)

Main result

The study found that "students' competencies in Master's thesis writing improves when they engage with our learning methodology, which incorporates generative AI (GenAI) training. The 2024-2025 cohort demonstrated a higher mean grade (16.49 vs. 15.55) and a shift in the median from 15 to 18, indicating a substantial improvement in performance." Additionally, "students that experimented the learning methodology expressed a need for additional support to optimize their use of GenAI tools," and the analysis revealed that students without GenAI training showed declining performance and increased misuse of these tools.

Research paradigm

Design Science Research (DSR) / pragmatist

Author conclusions

The authors conclude that "through this study, we designed and implemented a specialized learning methodology that intentionally cultivates AI literacy, reflective practice, awareness of the limitations and biases inherent in generative AI (GenAI) systems, academic integrity, and critical thinking skills." They further state: "The evaluation of our artifact reveals that students engaged with this learning methodology significantly improved their competencies in Master's thesis writing. Conversely, it also highlights that students who lack training in GenAI use for thesis writing are at greater risk of failing this task. This finding opens new avenues for research by proposing a framework for a learning methodology that leverages generative AI effectively to support students' competence development."

Risk of bias

Selection bias: Study limited to Information Systems Management students with strong technological backgrounds; Confounding variables: Different cohorts (MSI vs. CMSI) have different academic backgrounds and prior Master's degree completion status; Lack of random assignment: Cohorts were naturally occurring groups rather than randomized; Observer bias: Professors evaluating theses, though blinded to methodology participation, may have developed enhanced detection of GenAI misuse over time; Attrition: No explicit mention of dropout rates or missing data; Selection bias: Participants are Information Systems Management students with strong technological background, not representative of broader student population; Cohort composition differences: CMSI students already hold a Master 2 degree whereas MSI students are first-time thesis writers, potentially affecting comparability; Grader knowledge: While professors were described as 'blinded,' the paper does not explicitly confirm they were unaware of cohort identity during grading; Small sample sizes: Cohort 1a (n=29), Cohort 1b (n=25), Cohort 2a (n=15), Cohort 2b (n=21); Confounding variables: Multiple factors changed between 2023-2024 and 2024-2025 beyond just GenAI training; Self-report bias: Survey and interview data subject to social desirability bias; Attrition/non-response: No reporting of survey or interview response rates; Selection bias: Only Information Systems Management students participated; no diverse disciplinary representation; Potential confounding variables: Differences in academic background between MSI and CMSI students (CMSI students already held Master 2 degrees); Hawthorne effect: Students aware they were part of a study may have modified behavior; Grading bias: Although professors were blinded to methodology, their ability to detect GenAI misuse may have evolved between academic years; Attrition/missing data: No explicit reporting of response rates for surveys and interviews; Cohort 2b declining performance unexplained: Could reflect instructor factors rather than methodology

Limitations

The study was "conducted with Information Systems Management students, a population characterized by a strong technological background
While these findings provide valuable insights, a promising avenue for future research lies in testing this learning environment with students from diverse academic disciplines." Additionally, the authors note that "the qualitative testimonies provided by students in Cohort 1b would benefit from "a systematic examination of these accounts" through thematic analysis, and they lack "in-depth interviews with professors who evaluated these theses" to provide complementary perspectives on GenAI's benefits and risks.

Open questions raised

Need for thematic analysis of student qualitative testimonies regarding GenAI influence on specific competencies (critical thinking, argument structuring, originality)
Lack of mandatory reflective documents on GenAI use across all cohorts
Missing evaluation of students' ability to select and apply GenAI tools appropriately at each research stage
Absence of in-depth interviews with professors evaluating theses regarding GenAI benefits, risks, and detection strategies
Limited exploration of GenAI integration with specialized tools (Pro versions) versus free tools
Lack of research with non-technical discipline students (humanities, social sciences) to assess transferability of methodology

Extracted from: pdfAgreement 65%

Explore related topics

AI Manuscript Writing Doctoral Training Design Science