AI evidence extraction

Scaffolding systematic reviews in learning design and technology through mentoring and AI integration

Xiyu Wang, Fatemeh Dadashipour, Basori, Yukiko Maeda, Jennifer C. Richardson · Educational Technology Research and Development · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

3/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1007/s11423-026-10629-8

Methodology & findings

Study design

This is a practice-informed reflective analysis based on the authors' experience conducting a systematic review on learning analytics in higher education as a multidisciplinary team with novice and experienced researchers.

Sample

N = 242, 2 groups

Primary method

Latent Dirichlet Allocation (LDA) topic modeling (Blei et al., 2003) applied to titles and abstracts of 242 published reviews. Thematic frequency analysis provided quantitative summary of topic prevalence. Qualitative analysis of case study experience (authors' own SR) through reflective narrative synthesis organized by review stages.

Main result

The paper identifies that "SRs are becoming increasingly popular across education and social science research contexts" and documents that among 242 published reviews in 11 top-tier LDT journals between 2011-2025, eight prominent thematic areas emerged: "(1) Computational tools, robotics, and programming (21.7%, n = 53), (2) adaptive and individualized instructional design (14.8%, n = 36), (3) Online learning, learner support, and collaboration (12.7%, n = 31), (4) Student learning outcomes and assessments (11.07%, n = 27), (5) Immersive and virtual learning environments (11.07%, n = 27), (6) Game-based learning, gamification, and pedagogical design (10.25%, n = 25), (7) Applications of technology in teaching and instructional design (10.25%, n = 25), and (8) Generative AI integration in education (8.2%, n = 20)." The key learning finding is that "SR method is most effectively learned through practice" and that "mentoring functioned as an integrated support system that combined methodological guidance with emotional scaffolding."

Reports effect sizes.

Research paradigm

interpretivist/pragmatist

Author conclusions

The authors conclude that "conducting an SR is not only a methodological exercise but also an inherently iterative process that calls for robust technical strategies, careful interpretive judgment, and, at times, emotional resilience" and emphasize that "Establishing mentoring structures, balancing automation with human judgment, and cultivating a supportive and collaborative environment are as essential as mastering procedures of SR." They further state that "SR methodological skills are best developed through experiential learning supported by mentorship and collaboration." Regarding automation, they note that "these cannot yet replace the interpretive rigor that human researchers bring in the process" and that "Selective, judicious use of technology, coupled with careful verification by humans, can help reduce repetitive, time-consuming workload for efficiency while safeguarding credibility and integrity of the review process."

Risk of bias

The paper's findings are based on a single team's experience conducting one systematic review, potentially limiting generalizability; The reflective nature of the analysis may introduce confirmation bias regarding the effectiveness of described strategies; The mapping analysis of 242 reviews uses LDA topic modeling, which may miss relevant studies or misclassify thematic areas; The interdisciplinary focus on LDT may not fully represent systematic review practices in other fields with different publication norms; Selection bias in review catalog: Limited to 11 top-tier journals, may miss relevant syntheses in other outlets; Sampling bias in literature: Authors acknowledge "sampling errors, in this case, the omission of relevant studies, can occur, potentially leading to biased conclusions"; Expertise bias: Case study reflects experiences of specific interdisciplinary team; generalizability unclear; Automation tool evaluation: Testing of AI tools (AIScreenR, MetaMate) was exploratory and not systematically compared; Publication bias: Analysis covers published reviews only, not protocols or abandoned reviews

Open questions raised

Limited practical, hands-on, experience-based guidance for novice SR researchers, especially emerging scholars
Uneven advancement in automation across different SR stages (screening has advanced most; data extraction, synthesis, and reporting remain heavily dependent on human expertise)
Under-discussion of sampling strategies to reduce sampling bias in systematic reviews
Need for guidance specific to conducting SRs in interdisciplinary fields like LDT where literature spans diverse disciplinary domains
Need to understand how to wisely integrate automation tools in human review processes to balance efficiency and credibility
Need for hands-on, experience-based guidance for novice SR researchers, particularly in interdisciplinary fields like LDT

Code: R scripts mentioned for data handling and analysis but no repository URL provided. Authors reference development of "R scripts to handle standard analytic tasks, for example generating descriptive statistics for each item and reshaping the data into summary tables."Extracted from: pdfAgreement 57%

Explore related topics

Systematic Review Automation Literature Discovery Case Study