AI evidence extraction

Artificial Intelligence in Higher Education

Bruno Poëllhuber, Normand Roy, Alexandre Lepage · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

2.82

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1201/9781003320791-17

Methodology & findings

Study design

Narrative literature review and conceptual analysis.

Main result

The paper identifies three major areas of AI application in higher education with distinct opportunities and limitations. The authors found that "there is a great deal of work on predictive analytics and dashboard development, and research results on student engagement and retention seem positive." However, they also note that "there is still a lack of large-scale, rigorous research examining the effectiveness of LAK for student success." For adaptive learning, the authors conclude that "AI can do it faster and in a more individualized way as long as the learner's model is sufficiently complex," but such systems "reveal serious limitations" for higher-order learning objectives. Regarding generative AI, they observe that "ChatGPT fails to apply the steps of logical reasoning adequately and remains at a very superficial level on many complex subjects," yet these tools "have the potential to reduce certain types of interactions with humans" while "could also make room for higher levels in class discussions."

Research paradigm

Critical interpretivism with human-centered design orientation

Author conclusions

The authors conclude that "Will AI eventually be able to replace teachers? Our human-centered approach says a clear 'No.'" They emphasize that "the role of teachers will be certainly be questioned and transformed, and, in some cases—in courses that focus on knowledge transmission or in domains where there is little advancement— the teachers' role may be threatened." Most importantly, they assert that "AI literacy will also allow these actors to participate in the discussion of its social, ethical, legal, and economic impacts, in order to make relevant, thoughtful, and ethical use of it." They conclude that "Sound knowledge of these tools, their functioning, their limits, and their biases will be crucial for relevant and ethical human-centered use."

Risk of bias

Selection bias in LAK research - most studies focus on proofs of concept rather than rigorous implementations; Data bias in generative AI - ChatGPT is trained on "Internet prior to 2021, a body of work that is biased in many ways and in which many topics have been subject to major disinformation campaigns"; Potential Hawthorne effect in learning analytics dashboards where teacher perceptions become self-fulfilling; Risk of false positives and false negatives in dropout prediction models; Potential bias against students with effective study strategies that don't involve many clicks; The paper discusses multiple systemic biases in AI systems: biases in training data (particularly for ChatGPT which "feeds on what it found on the Internet prior to 2021, a body of work that is biased in many ways"), risks of LAK models incorrectly identifying at-risk students based on click behavior rather than actual learning, potential for Hawthorne effect and self-fulfilling prophecies in student interventions, and societal biases embedded in AI systems generally.

Limitations

The authors acknowledge that for learning analytics, "there is still a lack of large-scale, rigorous research examining the effectiveness of LAK for student success and the characteristics that might drive adoption and uses." They further note that "the adoption of dashboards still seems to be problematic, and explicit links to suggested actions (for students) or interventions (for teachers) are still missing in most projects." Regarding adaptive learning implementation, "Pedagogical challenges relate to the identification of what can and cannot be learned through adaptive learning, as well as to the required self-regulation abilities that students must demonstrate." For generative AI, they state that "ChatGPT and other generative AI tools should not be seen as reliable, since they do not distinguish between true and false information and, even worse, can literally propose fabricated answers and even invent false references."

Open questions raised

Lack of large-scale, rigorous research examining the effectiveness of LAK for student success
Need for understanding which learning outcomes can and cannot be achieved through adaptive learning
Limited research on the characteristics that drive adoption and use of LAK systems in higher education institutions
Need for explicit links to suggested actions in dashboard design for both students and teachers
Research examining pedagogical applications of generative AI beyond knowledge transmission
Studies on the effectiveness of AI literacy training for educators

Extracted from: pdfAgreement 69%

Explore related topics

Human-AI Collaboration Conceptual