AI evidence extraction

Systematic review of research on artificial intelligence applications in higher education – where are the educators?

Olaf Zawacki‐Richter, Victoria I. Marín, Melissa Bond, Franziska Gouverneur · International Journal of Educational Technology in Higher Education · 2019

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

6/10

Relevance

2/4

Quality (LMQS)

Evidence

5,282

Citations

138.64

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s41239-019-0171-0

Methodology & findings

Study design

Systematic review using explicit inclusion/exclusion criteria.

Main result

The systematic review identified 146 articles on AI in higher education from 2007-2018. The authors found that "the majority of the disciplines involved in AIEd papers come from Computer Science and STEM, and that quantitative methods were the most frequently used in empirical studies." Four main areas of AI application were identified: "1. profiling and prediction, 2. assessment and evaluation, 3. adaptive systems and personalisation, and 4. intelligent tutoring systems." A critical finding was that "only five out of 146 included articles (3.4%) provide an explicit definition of the term 'Artificial Intelligence'" and "only two out of 146 articles (1.4%), critically reflect upon ethical implications, challenges and risks of applying AI in education."

Research paradigm

Interpretive/critical realist - systematic synthesis of existing literature to map research landscape and identify gaps

Author conclusions

"A stunning result of this review is the dramatic lack of critical reflection of the pedagogical and ethical implications as well as risks of implementing AI applications in higher education." The authors conclude that "More research is needed from educators and learning designers on how to integrate AI applications throughout the student lifecycle, to harness the enormous opportunities that they afford for creating intelligent learning and teaching systems. The low presence of authors affiliated with Education departments identified in our systematic review is evidence of the need for educational perspectives on these technological developments." They emphasize that "educational technology is not (only) about technology it is the pedagogical, ethical, social, cultural and economic dimensions of AIEd we should be concerned about."

Risk of bias

Language bias (English/Spanish only); publication bias (peer-reviewed journals only, excluding conference proceedings and grey literature); database selection bias (three databases may not capture all relevant publications); temporal bias (2007-2018 cutoff); author disciplinary affiliation bias noted (62% from Computer Science/STEM, only 8.9% from Education departments suggesting potential research gap).; Language bias - exclusion of non-English/Spanish publications; Publication bias - exclusion of conference proceedings and grey literature; Database coverage bias - limited to three databases, potentially missing indexed journals not covered; First-author affiliation bias - geographical and disciplinary representation based on first author only; Language bias: Only English and Spanish articles included; Publication bias: Only peer-reviewed journal articles; conference proceedings and grey literature excluded; Database selection bias: Only three databases searched (EBSCO, Web of Science, Scopus); Geographic bias: 50% of articles from four countries (USA, China, Taiwan, Turkey); Disciplinary bias: 62% of first authors from Computer Science/STEM departments; only 8.9% from Education; Temporal bias: Study period 2007-2018 may not capture recent developments; Author affiliation bias: Dominance of Computer Science perspectives over educational perspectives; Language bias: exclusion of non-English and non-Spanish publications; Database bias: limited to three international databases; exclusion of conference proceedings, books, grey literature; Publication bias: peer-reviewed journals only; Geographic bias: 50% of articles from only four countries (USA, China, Taiwan, Turkey); Disciplinary bias: 62% of authors from Computer Science and STEM departments; only 8.9% from Education backgrounds; Selection bias: requirement for primary empirical/descriptive research excluded review articles and other publication types

Limitations

"Although the three educational research databases chosen are large and international in scope, by applying the criteria of peer-reviewed articles published only in English or Spanish, research published on AI in other languages were not included in this review
This also applies to research in conference proceedings, book chapters or grey literature, or those articles not published in journals that are indexed in the three databases searched." The authors also note that "although Spanish peer-reviewed articles were added according to inclusion criteria, no specific search string in the language was included, which narrows down the possibility of including Spanish papers that were not indexed with the chosen keywords."

Open questions raised

Major gaps identified include: (1) lack of critical reflection on ethical implications and risks in 98.6% of articles; (2) weak connection to theoretical pedagogical perspectives; (3) limited research from educators (only 8.9% first authors from Education departments); (4) lack of longitudinal studies; (5) need for design-based approaches rather than purely technological pilot studies; (6) insufficient attention to privacy and data protection concerns; (7) underexplored potential of AI for collaborative learning; (8) limited teacher-facing AI research despite significant potential; (9) need for explicit pedagogical theory underpinning empirical studies; (10) need for exploration of ethical and educational approaches in AIEd implementation.
Lack of longitudinal studies - most studies are short-term or pilot studies
Absence of critical reflection on ethical implications and risks of AIEd (only 1.4% of papers address this)
Weak connection to theoretical pedagogical perspectives - majority of research atheoretical
Low presence of authors from Education departments (only 8.9% of papers)
Need for design-based research approaches beyond descriptive and technological studies

Data: Bibliography of included studies available: "A file with all included references is available at: https://www.researchgate.net/publication/335911716_AIED-Ref (CC-0; DOI: https://doi.org/10.13140/RG.2.2.13000.88321)"; Bibliography of included studies; The bibliography of included studies (146 articles) is available from corresponding author upon request. A file with all included references is available at: https://www.researchgate.net/publication/335911716_AIED-Ref (CC-0; DOI: https://doi.org/10.13140/RG.2.2.13000.88321); A file with all included references (146 articles) is available at: https://www.researchgate.net/publication/335911716_AIED-Ref (CC-0; DOI: https://doi.org/10.13140/RG.2.2.13000.88321)Code: No code repositories mentioned. Systematic review used EPPI Reviewer software for screening and coding.; None mentioned.Extracted from: pdfAgreement 39%

Explore related topics

Scholarly Infrastructure