AI evidence extraction

Artificial intelligence in higher education: the state of the field

Helen Crompton, Diane Burke · International Journal of Educational Technology in Higher Education · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

8/10

Relevance

3/4

Quality (LMQS)

Evidence

1,378

Citations

48.81

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s41239-023-00392-8

Methodology & findings

Study design

Systematic review using PRISMA principles and protocol.

Sample

N = 138, 1 group

Primary method

Deductive coding with a priori predetermined codes; inductive grounded coding using constant comparative method; inter-rater reliability calculated using percentage agreement (95% agreement initially, 100% after discussion); quantitative aggregation of data presented as whole numbers and percentages; no inferential statistics reported.

Main result

The findings of this study show that "in 2021 and 2022, publications rose nearly two to three times the number of previous years." Additionally, "the trend has shifted from the US to China leading in the number of publications." The study identified "five usage codes emerged from the data: (1) Assessment/Evaluation, (2) Predicting, (3) AI Assistant, (4) Intelligent Tutoring System (ITS), and (5) Managing Student Learning."

Reports effect sizes.

Research paradigm

Mixed methods (deductive and inductive); grounded theory approach with quantitative aggregation

Author conclusions

Five usage codes emerged from the data: (1) Assessment/Evaluation, (2) Predicting, (3) AI Assistant, (4) Intelligent Tutoring System (ITS), and (5) Managing Student Learning. The findings of this study provide a springboard for future academics, practitioners, computer scientists, policymakers, and funders in understanding the state of the field in AIEd HE, how AI is used. It also provides actionable items to ameliorate gaps in the current understanding. As the use AIEd will only continue to grow this study can serve as a baseline for further research studies in the use of AIEd in HE.

Risk of bias

Publication bias - only peer-reviewed journal articles included; Language bias - only English language articles included; Selection bias - hand search limited to specific journals that may reflect author preferences; Accessibility bias - studies from low-income countries underrepresented (only 1% from South America, 2% from Africa); Convenience sampling bias - researchers more likely to study their own student populations; Institutional bias - higher ease of ethical approval in HE vs K-12 may influence research focus; Publication bias: only peer-reviewed journal articles included; proceedings and other publication types excluded; Language bias: only English-language articles included; Database selection bias: specific databases and journals selected may not capture all relevant literature; Selection bias in screened articles: 237 articles excluded during screening process; potential for systematic exclusion patterns; Researcher affiliation coding: limited to first author only, potentially missing interdisciplinary collaboration patterns; Accessibility bias: undergraduate population overrepresented at 72% due to convenience sampling in HE research; Publication bias (only peer-reviewed journal articles included, excludes conference proceedings and editorials); Language bias (only English-language articles included); Selection bias in hand search (limited to specific journals); Accessibility bias for HE researchers (easier access to undergraduate populations); Researcher familiarity bias (both researchers may have shared perspectives on AIEd); Time period limitation (2016-2022 may miss earlier foundational work); Selection bias: Only peer-reviewed journal articles included; conference proceedings and editorials excluded, potentially missing recent developments; Publication bias: Articles indexed in major databases may not represent all AIEd research globally; Researcher accessibility bias: HE researchers more likely to study their own accessible student populations; Convenience sampling bias: Larger undergraduate populations more readily available for study than graduate populations

Limitations

The paper states that "there do not appear to be any studies examining the last 2 years of AIEd in HE" in prior reviews, indicating a temporal gap
Additionally, regarding the focus on students, the authors note: "However, this large number of student studies in HE may be due to the student population being more easily accessibility to HE researchers who may study their own students
The ethical review process is also typically much shorter in HE than in K-12." The high focus on undergraduate students (72%) is noted as potentially due to convenience sampling.

Open questions raised

The authors identify gaps for future research including: 1) Need for more research on graduate students (only 9% of studies); 2) Increased focus on instructor use of AI (only 17%); 3) Greater emphasis on manager/administrative use of AI (only 11%); 4) Research on new tools such as "Chat GPT"; 5) Attention to ethical implications and privacy concerns regarding predictive analytics; 6) Further investigation of AI applications in underrepresented continents (South America 1%, Africa 2%); 7) Research examining how AI transforms learning versus replicating past practices.
Limited research on graduate students (only 9% of studies); graduate education AIEd needs underexplored
Underrepresentation of research in low-income countries (Africa 2%, South America 1% of studies)
Limited focus on managers/administrators (11% of studies) despite AI's potential for data-driven decision making across large datasets
New tools such as ChatGPT not yet examined in the literature reviewed (through 2022)
Limited pedagogical and ethical implications of implementing AI in HE (noted in prior work)

Data: "The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request."; The authors state: "The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request." No public repository or direct URL provided.; The datasets used and/or analysed during the current study are available from the corresponding author (crompton@odu.edu) on reasonable request.; The authors state: "The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request."Extracted from: pdfAgreement 46%

Explore related topics

Responsible AI Use AI Governance