AI evidence extraction

Generative AI and higher education: a review of claims from the first months of ChatGPT

Lasse X Jensen, Alexandra Buhl, A. V. N. L. Sharma, Margaret Bearman · Higher Education · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

9.53

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1007/s10734-024-01265-3

Methodology & findings

Study design

Scoping review methodology combined with critical thematic analysis.

Main result

The analysis identified ten claims about ChatGPT and higher education across three categories: claims about ChatGPT's nature, claims about changing institutional and teaching practices, and claims about new student practices. The authors found that "Overall, the claims present a positive perspective on AI in higher education. While being perceived as a disruption of the status quo, the authors generally frame AI as a catalyst for existing agendas, e.g. assessment reform, personalisation, or inclusion." The study reveals that "the claims mainly portray students as either plagiarists or victims of a failing educational system," with limited discussion of productive student uses of AI.

Research paradigm

Interpretive/hermeneutic (critical discourse analysis of grey literature)

Author conclusions

"This analysis of the early grey literature on ChatGPT alerts us to claims about the technology and its role in the future of higher education. While the examined literature contains some caveats about its possible negative influences, particularly concerning the decoupling of thinking and writing, the majority of articles were highly positive. As we move forward and improve our understanding of genAI and its effect on teaching and learning, those working in higher education may use this synthesis to be mindful of how our early assumptions can influence what and how we teach and research." The authors warn against over-optimism and emphasize the need for deeper critical consideration of risks and ethical considerations, interrogation of existing educational agendas, and ensuring students are part of conversations about generative AI.

Risk of bias

Selection bias: Only five English-language higher education sector outlets were searched; many discussions occurred on Twitter and webinars; Geographic/linguistic bias: 25 of 45 articles authored from the USA alone; only one article from a low- or middle-income country (South Africa); Temporal bias: Analysis limited to first 3 months post-launch; may not reflect more mature perspectives; Publication source bias: Grey literature (opinion pieces, blogs) rather than peer-reviewed research; Novelty effect: Authors acknowledge tendency for new technologies to be judged more positively simply because they are new; Hype cycle effect: Early inflated expectations typical of new technologies; Selection bias: articles identified only from five major English-language sector outlets, excluding conversations on Twitter, webinars, and local university newspapers; Geographic/linguistic bias: 25 of 45 articles authored by authors based in USA; only one author from low- or middle-income country (South Africa); Source type bias: focus on grey literature (opinion pieces, blog posts) rather than peer-reviewed scholarship, which may reflect different perspectives; Temporal bias: early adoption period (first 3 months) may capture inflated expectations due to novelty effect and hype cycle; Author bias: nearly all authors employed in higher education sector, potentially reflecting institutional perspectives; Selection bias: Five sector outlets searched; conversations in Twitter, webinars, and other platforms excluded; Geographic bias: 25 of 45 authors from USA; only one author from low/middle-income country (South Africa); Language bias: English-language outlets only; Temporal bias: First 3 months only; may not capture more mature perspectives; Author background bias: Almost all authors employed in higher education sector; limited diversity of perspectives

Limitations

"The main limitation to consider relates to the way we identified the articles
Searching sector outlets captures only a subset of perspectives
In the first months after ChatGPT's release, many conversations took place in other spaces, such as Twitter or various online webinars
Furthermore, owing to our search strategy, the included papers have a strong bias towards authors from anglophone high-income countries
This means that most of the perspectives described in this review reflect fairly similar traditions of teaching and learning." The 3-month timeframe may have captured less mature perspectives than a longer observation period would provide.

Open questions raised

Gap between broad-scale concerns and fine-grained research that responds to them: While emerging empirical literature focuses on micro-issues (e.g., distinguishing ChatGPT writing from human writing), broader concerns about thinking-writing decoupling and ethical engagement are underexplored
Lack of empirical evidence on actual impacts: Most early claims are speculative rather than grounded in empirical research
Student voice absent: Claims about how students may productively use genAI were least prevalent (21 of 45 articles); more research involving students directly is needed
Limited perspectives from non-English-speaking countries and low/middle-income contexts
Need for more mature perspectives beyond the initial 3-month period
The authors identify a gap between broad-scale concerns mentioned in the literature and fine-grained empirical research responding to them. They note that while emerging empirical literature focuses on micro-issues (e.g., distinguishing ChatGPT-written text from human writing), there is limited empirical research addressing the major concerns raised. Additionally, the authors recommend deeper involvement of students in discussions about generative AI in higher education, noting that students are primarily portrayed as either plagiarists or victims rather than active participants in shaping how these technologies are used.

Data: Supplementary materials table provided with article title, author name, author location, outlet, publication date, and word count for each of the 45 included articles (available at https://doi.org/10.1007/s10734-024-01265-3).; Supplementary materials provided at https://doi.org/10.1007/s10734-024-01265-3, including a table with overview of article title, author name, author location, outlet, publication date, and word count for each of the 45 included articles.; Supplementary materials referenced as available online at https://doi.org/10.1007/s10734-024-01265-3, including a table with article details (title, author name, author location, outlet, publication date, word count) for each of the 45 included articles.Extracted from: pdfAgreement 43%

Explore related topics

AI Manuscript Writing Academic Writing Literature Review