AI evidence extraction

A comprehensive AI policy education framework for university teaching and learning

Cecilia Ka Yuk Chan · International Journal of Educational Technology in Higher Education · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

1/4

Quality (LMQS)

Evidence

1,160

Citations

41.01

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1186/s41239-023-00408-3

Methodology & findings

Study design

Mixed-methods survey design combining quantitative (closed-ended questions with descriptive analysis) and qualitative (open-ended questions with thematic analysis) data collection from 457 undergraduate and postgraduate students and 180 teachers and staff across various disciplines in Hong Kong universities via online questionnaire..

Sample

N = 637, 3 groups

Primary method

Descriptive analysis (means, standard deviations, medians) for quantitative survey data. Thematic analysis using an inductive approach for qualitative open-ended responses, where themes emerged from the data rather than being predetermined. No inferential statistical tests reported.

Main result

The study found that "both students (mean = 2.28, SD = 1.18) and teachers (mean = 2.02, SD = 1.1) reported relatively low experience" with generative AI technologies, yet "both groups demonstrated a belief in the positive impact of integrating AI technologies into higher education (students: mean = 4, SD = 0.891; teachers: mean = 3.87, SD = 1.32)." Additionally, "both groups expressed doubt about teachers' ability to accurately identify a student's usage of generative AI technologies for completing assignments (students: mean = 3.02, SD = 1.56; teachers: mean = 2.72, SD = 1.62)."

Reports effect sizes.

Research paradigm

Mixed-methods (pragmatism)

Author conclusions

The authors conclude that "This study proposes an AI Ecological Education Policy Framework to address the diverse implications of AI integration in university settings. The framework consists of three dimensions—Pedagogical, Governance, and Operational—each led by a responsible party. This structure allows for a more comprehensive understanding of AI integration implications in teaching and learning settings and ensures stakeholders are aware of their responsibilities. By adopting this framework, educational institutions can align actions with their policy, ensuring responsible and ethical AI usage while maximizing potential benefits."

Risk of bias

Selection bias: convenience sampling based on availability and willingness; Self-report bias: reliance on self-reported data from participants; Sample representativeness: relatively small sample may not represent all educational institutions; Scope limitation: only text-based generative AI (ChatGPT, Bing, Co-Pilot) examined; Selection bias: Convenience sampling method based on availability and willingness to participate, not random selection; Self-report bias: Data collected through self-reported survey responses; Sample representativeness: Small sample size (457 students, 180 staff) may not represent all educational institutions; Temporal limitation: Study conducted during early adoption phase of ChatGPT; findings may not generalize to later periods; Geographic limitation: Data collected only from Hong Kong universities, limiting generalizability; Selection bias from convenience sampling method based on availability and willingness to participate; Self-report bias from reliance on self-reported survey data; Sample representativeness: potentially small and non-representative sample from Hong Kong universities only; Temporal bias: UNESCO framework used as basis was developed before GPT-3.5 and GPT-4 availability

Limitations

"This study has some limitations, including a relatively small sample size that may not be representative of all educational institutions
Additionally, the research only focused on text-based generative AI technology and did not explore other types or variations
Lastly, the study relied on self-reported data from participants, which may be subject to bias or inaccuracies."

Open questions raised

The authors note that more research is necessary to fully comprehend AI integration in higher education. They highlight that while UNESCO's recommendations provide high-level guidelines, they were formulated before the availability of GPT 3.5 and 4, and do not specifically address university teaching and learning contexts. The authors also note that existing AI education policies tend to emphasize instrumental workforce development but lack focus on transformative potential and ethical considerations.
More research is necessary to fully comprehend the implications of AI integration in higher education
Further investigation needed into other types and variations of generative AI technology beyond text-based models
Need for longitudinal studies to assess the long-term impacts of AI integration on student learning outcomes
Requirement for implementation and evaluation studies of the proposed AI Ecological Education Policy Framework in actual institutional settings
The authors identify that "more research is necessary to fully comprehend the" implications of AI integration. They note that existing AI education policies are "mostly generic and implicit because of the lack of concrete evidence of implementing AI technologies." The study suggests future work should explore types and variations of generative AI beyond text-based tools and examine implementation of the proposed framework.

Extracted from: pdfAgreement 63%

Explore related topics

Institutional Policy AI Governance Mixed Methods