AI evidence extraction

Practical and ethical challenges of large language models in education: A systematic scoping review

Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martínez‐Maldonado, Guanliang Chen et al. · British Journal of Educational Technology · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

1/4

Quality (LMQS)

Evidence

699

Citations

24.76

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1111/bjet.13370

Methodology & findings

Study design

Systematic scoping review of 118 peer-reviewed papers published since 2017, with structured assessment of practicality and ethicality using established frameworks across seven important aspects.

Sample

N = 118, 3 groups

Primary method

Systematic scoping review methodology with structured assessment framework across seven important aspects for evaluating practicality and ethicality.

Main result

The study identified "53 use cases for LLMs in automating education tasks, categorised into nine main categories: profiling/labelling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation." Additionally, the review revealed "several practical and ethical challenges, including low technological readiness, lack of replicability and transparency and insufficient privacy and beneficence considerations."

Reports effect sizes.

Research paradigm

Pragmatist/Mixed-methods (systematic evidence synthesis approach)

Author conclusions

The authors conclude that "As the intersection of AI and education is continuously evolving, the findings of this study can serve as an essential reference point for researchers, allowing them to leverage the strengths, learn from the limitations, and uncover potential research opportunities enabled by ChatGPT and other generative AI models." They further recommend "Adopting a human‐centred approach throughout the developmental process could contribute to resolving the practical and ethical challenges of large language models in education."

Risk of bias

Publication bias (peer-reviewed papers only, may exclude grey literature); Language bias (papers may be English-language focused); Time-bound search (since 2017 may exclude foundational work); Selection bias in study inclusion criteria; Potential publication bias (only peer-reviewed papers included); Temporal bias (papers from 2017 onwards only); Selection bias in database search strategy; Language bias (likely English-language papers only, though not explicitly stated); Researcher bias in categorization and assessment of practical/ethical challenges

Open questions raised

Need for updating existing innovations with state-of-the-art models (GPT-3/4)
Lack of open-sourcing of models and systems
Insufficient adoption of human-centred approaches in development
Need for improved reporting standards in empirical research on educational LLM technologies
Opportunities enabled by ChatGPT and other generative AI models remain underexplored
The authors identify the need for: (1) updating existing innovations with state-of-the-art models (e.g., GPT-3/4); (2) embracing open-sourcing of models/systems; (3) improving reporting standards for empirical research developing educational technologies using LLMs; and (4) adopting human-centred approaches in development processes.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 66%

Explore related topics

AI Manuscript Writing Multi-Domain Literature Review