AI evidence extraction

A Discipline-Agnostic AI Literacy Course for Academic Research: Architecture, Pedagogy, and Implementation

Gideon K. Gogovi · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Single-cohort, pre-post survey design with no control condition.

Primary method

Design-based research in education; scaffolded learning design grounded in Vygotsky's zone of proximal development and threshold concepts framework

Main result

The course produced self-reported outcomes consistent with its theoretical intentions. "The three largest and most statistically robust gains are theoretically significant in light of the course's design. Hallucination detection showed the largest gain among the eight confidence items (∆M = +1.20, d = +1.45, p < .001), a very large effect reflecting the course's sustained, module-spanning emphasis on recognizing and verifying AI failure modes. Responsible AI use showed the second largest gain (∆M = +1.03, d = +1.33, p < .001), consistent with the course's ethics-as-subject infrastructure." The very large effect sizes for AI attribution practice (d = +2.40) and hallucination detection (d = +1.45) were particularly noteworthy.

Research paradigm

Design-based research in education; constructivism with scaffolded learning framework

Author conclusions

"BSTA 495/395 demonstrates that AI research literacy can be developed as a coherent, teachable competency within standard higher education structures, delivered to mixed-level, multi-disciplinary cohorts without prerequisites, and organized around the authentic cognitive demands of literature review rather than abstract AI principles. Its four-module architecture, scaffolded skill progression, ethics-as-subject infrastructure, and process-centered assessment system constitute a replicable design model for institutions seeking to develop comparable offerings." The authors note that "The students who complete this course leave with a replicable workflow for AI-assisted research, the critical habits of mind to apply it responsibly across a changing tool landscape, and the documented experience to represent their AI use transparently in professional and scholarly contexts."

Risk of bias

Selection bias: single institution, single instructor, single cohort; Attrition: 27 pre-survey respondents, 29 post-survey respondents (n=26 matched pairs); No control condition: cannot separate course effects from maturation, regression to mean, or general AI tool exposure; Self-report bias: confidence and skills measured via self-assessment rather than independent verification; Ceiling effects: students entering with higher pre-course confidence (reading papers, methodology evaluation, limitations recognition) showed smaller gains; Social desirability bias: students may overreport learning gains in post-course surveys; Selection bias: Self-selected student enrollment in elective course; Maturation effects: Students gain experience naturally over 13 weeks regardless of instruction; Regression to the mean: Possible for confidence measures; Attrition bias: Some students withdrew after Week 1; n=26 for paired comparisons vs. n=27 pre-survey; Social desirability bias: Self-report confidence measures may reflect positive response bias; Lack of control group: Cannot establish causal attribution; Single instructor: No multi-instructor replication within study; Single institution: Limited generalizability; No control group; maturation and regression to the mean cannot be ruled out; Self-reported confidence as proxy for actual competency (not direct measure); Small sample size (n=26 for matched analyses) limiting statistical power and generalizability; Single instructor delivery limiting transferability of implementation; Ceiling effect on three competency items where students entered with above-average confidence; Attrition: some students withdrew after Week 1 post-survey administration; others enrolled after pre-survey

Limitations

"The course was developed and delivered by a single instructor in its inaugural offering at a single institution, limiting the evidence base for claims about effectiveness
Section 7 presents preliminary self-report outcome data from the Spring 2026 cohort
these findings are consistent with the design's intended effects but should be interpreted cautiously given the single-cohort, pre-post design without a control condition
Formal inferential analysis and a dedicated outcomes study with comparison data are ongoing." Additionally, "The 13-week format, while achievable within a standard academic calendar, is likely insufficient to fully develop the advanced synthesis competencies addressed in Module D." The course "also lacks explicit instruction in quantitative synthesis methods (meta-analysis, systematic review protocols, and PRISMA procedures)."

Open questions raised

Lack of formal inferential analysis and dedicated outcomes study with control conditions
Need for research examining whether the instructional inversion (centering tool use while making methodological knowledge visible) produces superior methodological knowledge acquisition compared to traditional approaches
Integration of quantitative synthesis methods (meta-analysis, systematic review protocols, PRISMA procedures) within AI-assisted workflow, particularly for health sciences and psychology
Extended Module D or two-semester sequence for students with limited prior research experience
Comparative study of the course's effectiveness across multiple institutions and instructors
Investigation of transfer of learned competencies to novel AI tools and contexts not directly taught in the course

Extracted from: pdfAgreement 57%

Explore related topics

AI Literacy Training Doctoral Training Design Science