AI evidence extraction

Generative AI tools and assessment: Guidelines of the world's top-ranking universities

Benjamin Luke Moorhouse, Marie Alina Yeo, Yuwei Wan · Computers and Education Open · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

8/10

Relevance

1/4

Quality (LMQS)

Evidence

343

Citations

12.16

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1016/j.caeo.2023.100151

Methodology & findings

Study design

Scoping review examining the content and coverage of assessment guidelines from the world's 50 top-ranking higher education institutions (HEIs) regarding generative AI use.

Sample

N = 50, 5 groups

Primary method

Qualitative content analysis of institutional guidelines. No quantitative statistical analysis reported. The review appears to employ thematic coding and categorization of guideline content into three main areas (academic integrity, assessment design advice, student communication).

Main result

The study found that "just under half of the institutions have developed publicly available guidelines." The guidelines primarily cover three main areas: academic integrity, advice on assessment design, and communicating with students. Notably, "Among the suggestions for teachers on assessment design, two appear particularly pertinent in helping develop effective assessment tasks and developing learners' AI-literacy: first, running assessment tasks through GAI to check the extent to which the tool can accomplish the task and, second, having students use GAI as part of the assessment process."

Reports effect sizes.

Research paradigm

Qualitative interpretivism with content analysis

Author conclusions

The authors conclude that "Overall, the review suggests that HEIs have come to accept the use of GAI and drafted assessment guidelines to advise instructors on its use." They further argue that "it may be beneficial to embrace GAI as a part of the assessment process since this is the reality of today's educational and job landscape. This will require instructors to develop a new competence - generative artificial intelligence assessment literacy - which is conceptualised in this article."

Risk of bias

Selection bias: Limited to top 50 ranked institutions only; Availability bias: Only publicly available guidelines were examined; Timeliness bias: Guidelines were developed rapidly in response to GAI emergence; Generalizability concerns: Findings may not apply to lower-ranked institutions; Selection bias: Only examined top 50 universities, not representative of all HEIs globally; Publication bias: Only included publicly available guidelines, missing internal or non-published policies; Geographic bias: Likely concentration in English-speaking institutions with published English-language guidelines; Temporal bias: Guidelines examined at a single point in time during rapid policy development phase; Selection bias: limited to top-50 ranked institutions only, potentially overrepresenting elite universities; Information bias: restricted to publicly available guidelines, missing internal or non-public guidance; Potential temporal bias: guidelines were rapidly developed in response to GAI release, may not reflect mature policies

Limitations

The review is limited to the world's 50 top-ranking institutions, which may not represent the broader landscape of higher education
The authors note that "these guidelines were developed in haste," which could affect the comprehensiveness and quality of the guidelines examined
The review focuses only on publicly available guidelines, potentially excluding institutions with internal guidance.

Open questions raised

The authors identify the need for instructors to develop a new competence termed 'generative artificial intelligence assessment literacy.' They also suggest that future work should focus on how to effectively integrate GAI into assessment rather than simply restricting its use.
The authors identify the need for institutions to develop generative artificial intelligence assessment literacy as a new competence for instructors. They suggest a paradigm shift from viewing GAI as a threat to academic integrity toward embracing it as an educational tool within assessed learning activities.
The review identifies the need for institutions to develop 'generative artificial intelligence assessment literacy' as a new instructor competence. It suggests that moving beyond restrictive policies toward embracing GAI in assessment design is an important direction for future practice and research.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 58%

Explore related topics

Responsible AI Use Peer Review Content Analysis