AI evidence extraction

ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?

Jürgen Rudolph, Samson Tan, Shannon Tan · Journal of Applied Learning & Teaching · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

2/4

Quality (LMQS)

Evidence

1,674

Citations

59.40

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.37074/jalt.2023.6.1.9

Methodology & findings

Study design

Desktop analysis approach combining extensive literature review (166 sources: 55% peer-reviewed academic sources, 45% non-academic sources) with experimental testing of ChatGPT through multiple queries to evaluate its functionality, strengths, and limitations.

Primary method

Qualitative content analysis of literature sources; descriptive analysis of ChatGPT output examples; narrative synthesis organized by AIEd framework (student-facing, teacher-facing, system-facing applications).

Main result

The study found that "ChatGPT can be beneficial in providing conceptual explanations and applications. However, the AI is less competent with content that requires higher-order thinking (critical, analytical thinking)." Additionally, "ChatGPT occasionally does hallucinate and spout nonsense, for instance, by inventing references," yet the authors conclude that "major changes to traditional higher education assessments such as essays and online exams are in order to address the existence of increasingly powerful AI."

Reports effect sizes.

Research paradigm

interpretivist/qualitative with exploratory experimentation

Author conclusions

The authors conclude that "ChatGPT could be the 'beginning of the end of all white-collar knowledge work' and 'a precursor to mass unemployment,'" but "whilst the alarmist and sensationalist reporting in news media is, in our view, not justified, it will be important to watch and engage in this fast-developing space and adjust learning, teaching, and assessment approaches in higher education." They further state: "We have seen that ChatGPT occasionally does hallucinate and spout nonsense... At the same time, we believe that major changes to traditional higher education assessments such as essays and online exams are in order."

Risk of bias

Selection bias in literature review: only 2 peer-reviewed articles available at time of writing, heavy reliance on preprints and grey literature; Temporal bias: knowledge cutoff of ChatGPT at 2021 limits current events assessment; Testing bias: authors conducted non-systematic 'random tests' rather than structured experimental design; Sample composition bias: testing limited to English and Chinese, not representative of all languages; Author bias: authors position themselves as advocates for educational transformation rather than neutral observers; Selection bias in literature review: limited peer-reviewed sources available at time of writing due to topic novelty; Observer bias in qualitative testing of ChatGPT outputs: interpretations of essay quality and language proficiency are subjective; Publication bias: reliance on non-academic sources (45% of sources) may skew representation of available evidence; Temporal bias: knowledge cutoff of ChatGPT (pre-2021) was not controlled as variable in testing; Non-systematic selection of test queries for ChatGPT; Limited peer-reviewed literature availability at time of writing; Authors' interpretive judgments in evaluating ChatGPT responses without standardized rubrics; Rapid evolution of technology rendering some findings potentially outdated; Single authors conducting testing without independent verification

Limitations

The authors note that "Due to the novelty of the topic, only about two peer-reviewed journal articles and eight preprints (academic papers that have not been peer-reviewed) on ChatGPT and higher education (especially on assessment, learning and teaching) were found by us as of 18 January 2023." Additionally, they acknowledge that their ChatGPT experiments involved only "a fraction of these random tests" and that "since ChatGPT is a brand-new product in the market, there is a dearth of empirical research to determine its implications on education."

Open questions raised

Lack of empirical research on ChatGPT's implications for higher education
Limited peer-reviewed literature on ChatGPT and assessment, learning and teaching (only 2 peer-reviewed articles found as of January 18, 2023)
Need for longitudinal studies on ChatGPT's long-term effects on student learning and academic integrity
Insufficient research on effective detection of ChatGPT-generated text and mitigation strategies
Limited exploration of ChatGPT in non-English language contexts and diverse educational systems
Need for research on proper integration of AI tools into curriculum design and pedagogical practice

Extracted from: pdfAgreement 67%

Explore related topics

AI Manuscript Writing Academic Writing Literature Review