AI evidence extraction

Thinking Along the Lines Generated by GenAI? A Systematic Mapping Study on Academic Writing

Éva Kaczkó, Lana Ivanjek, Lisa-Maria Norz, Elske Ammenwerth · Lecture notes in computer science · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1007/978-3-032-11108-1_35

Methodology & findings

Study design

Systematic mapping study following Petersen et al.'s methodological framework with adaptations by Tikva and Tambouris.

Main result

The mapping study revealed that "only nine studies were grounded in theories in academic writing, critical thinking, and/or pedagogy," and identified a critical shortage: "We identified a shortage of well-founded studies that conceptualize and investigate the relationship between GenAI, academic writing, and CT. Such studies are, however, essential for gaining deeper insights into how the lines generated by GenAI influence our thinking, and how we can address emerging challenges through purposeful pedagogical actions." The analysis shows that studies predominantly focused on cognitive skill definitions of critical thinking while overlooking broader transformative perspectives.

Research paradigm

Interpretivist/hermeneutic

Author conclusions

The authors conclude: "This mapping study provided a structured overview of current empirical research on the intersection of generative AI, academic writing, and CT in higher education. While GenAI tools are increasingly integrated into writing practices, theoretical grounding and pedagogically informed approaches remain scarce. The predominant focus on cognitive skill definitions of CT overlooks broader opportunities for transformation that are rooted in criticality and critical pedagogy. Future research should expand conceptual foundations, include educator perspectives, and develop empirically tested frameworks that support reflective and responsible AI use in academic writing."

Risk of bias

Language restriction to English-language peer-reviewed studies only; Database selection bias: limited to Scopus, Web of Science, EBSCO (exclusion of PsycInfo and grey literature); Temporal bias: restricted to 2023-2025 period, excluding foundational earlier work; Interpretative bias in theme categorization when studies did not explicitly define key concepts; Sampling bias in reliability checking: only sample of each coder's analyses cross-checked; Geographic bias: initial search yielded few studies from English-speaking countries, requiring broadened search; Publication bias: systematic reviews inherently subject to publication bias; no formal quality assessment conducted; Language restriction to English only may exclude relevant non-English publications; Limited temporal scope (2023-2025 only) may exclude foundational earlier work; Interpretative rather than standardized coding for theme analysis with only sample cross-checking; Lack of formal quality assessment of included studies limits ability to weight evidence strength; Search strategy dependent on selected databases (did not search PsycInfo or grey literature); Interpretative categorization of themes when studies lacked explicit definitions; Partial cross-checking of coded analyses (sample-based rather than comprehensive); Database and keyword selection bias in search strategy; Exclusion of grey literature, conference discussions, and other non-peer-reviewed sources; Language restriction to English only; Temporal restriction to 2023 onwards, potentially excluding foundational earlier work; Geographic bias toward English-speaking countries in broadened search

Limitations

The authors state: "the analysis and categorization of themes was primarily interpretative, particularly when studies did not explicitly define key concepts
For time reasons, we cross-checked only a sample of each coder's analyses (and resolved differing judgements through discussions)." Additional limitations include: "the results of any review are inherently dependent on the search strategies chosen, including the selection of databases, keywords, and inclusion/exclusion criteria
Searching other databases, such as PsycInfo, and other literature types, such as grey literature, conference discussions, blogs, and feuilletons, in which much of the discourse may also be unfolding, might have led to inclusion of more studies." Furthermore: "no formal quality assessment of the studies included was done, which limited our ability to weigh the strength of evidence" and "the review was restricted to literature published from 2023 onwards, reflecting the rapid evolution of the field following the public release of ChatGPT
While this temporal focus ensures relevance to the current technological landscape, it may have excluded earlier foundational work and thus narrows the historical perspective of the findings."

Open questions raised

Shortage of well-founded studies conceptualizing and investigating the relationship between GenAI, academic writing, and critical thinking with adequate theoretical grounding
Limited integration of educator/teacher perspectives (majority of studies focused on student perspectives)
Lack of clear conceptualization of critical thinking in the context of GenAI use in academic writing
Need for broader concepts from criticality and critical pedagogy movements to provide transformative perspectives
Absence of empirically tested pedagogical approaches and models that focus on pedagogy rather than technology
Limited research on discipline-specific approaches to critical thinking in academic writing

Extracted from: pdfAgreement 55%

Explore related topics

AI Manuscript Writing Academic Writing Meta-Analysis