AI evidence extraction

The Future of Feedback: Integrating Peer and Generative AI Reviews to Support Student Work

Akash K. Saini, Bill Cope, Mary Kalantzis, Gabriela C. Zapata · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

4.28

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.35542/osf.io/x3dct

Methodology & findings

Study design

Exploratory research design employing post-course surveys with both quantitative (Likert-scale ratings) and qualitative (open-ended questions) data collection; statistical analysis using paired t-tests and regression analysis; thematic analysis of open-ended responses.

Sample

N = 147, 4 groups

Primary method

STATA software employed for statistical analysis; paired samples t-tests to assess statistical significance between AI and peer review ratings; regression analysis to explore patterns and linear relationships between variables; dummy variable creation for non-numeric variables; descriptive and summary statistics; thematic analysis using Ryan and Bernard's (2003) framework with techniques including repetitions, similarities, differences, cutting and sorting, and word co-occurrence; data compiled in Microsoft Excel spreadsheet

Main result

Peer reviews were consistently perceived slightly higher across all three dimensions compared to AI reviews. The paired t-test reveals "a statistically significant difference in the mean quality ratings between AI and peer reviews (t = -3.11, p < 0.05). Participants, on average, rated peer reviews higher by approximately 0.21 units compared to AI reviews." Similarly for usefulness, "Participants, on average, found peer reviews more useful by approximately 0.40 units compared to AI reviews" and for actionability, "Participants, on average, rated peer reviews as more actionable by approximately 0.24 units compared to AI reviews." Additionally, "positive emotions, like curiosity and excitement, significantly boost the perceived quality, usefulness, and actionability of AI-assisted feedback."

Reports effect sizes.

Research paradigm

Mixed methods (positivist quantitative + interpretivist qualitative)

Author conclusions

"This research underscores the importance of integrating human expertise with AI technology in feedback mechanisms, offering practical insights for educators, instructional designers, and policymakers seeking to enhance feedback experiences through emerging digital technologies." Additionally, "By combining the expertise and empathy of peer review with the computational power and data-driven insights of AI models, this study makes a case for a dynamic and symbiotic feedback ecosystem facilitated by the emerging sub-disciplinary areas within the learning sciences: educational data mining and learning analytics."

Risk of bias

Selection bias: Voluntary survey participation without information on response rates; Social desirability bias: Self-reported perceptions of AI and peer feedback quality; Emotional response confounding: Participants' feelings toward AI appear interdependent with their ratings of quality, usefulness, and actionability; Sampling bias: Predominantly female (68.47%) and North American (85.75%) sample may not be representative; Selection bias: Voluntary participation in surveys without mention of response rate; Response bias: Self-reported perceptions may be influenced by emotional states and prior attitudes toward AI; Potential interdependence bias: Participants' feelings toward AI and their perceptions of feedback quality may be mutually reinforcing; Confounding: Emotional responses to AI may confound objective evaluation of feedback quality; Selection bias: Voluntary participation without random assignment; Self-selection bias: Students may have chosen to participate based on AI attitudes; Social desirability bias: Participants may have responded to please researchers regarding emerging AI technologies; Confounding: Emotional responses toward AI may confound perceived quality/usefulness/actionability ratings, as the authors note the potential 'interdependence'; Lack of temporal precedence: Cannot determine whether emotions shape perceptions or vice versa; No control group: Exploratory design without comparison to alternative feedback mechanisms; Attrition not reported: No discussion of survey completion rates or missing data mechanisms

Open questions raised

The authors identify the need for deeper investigation into how emotional responses toward AI influence reception of feedback and how participants distinguish utility of AI-generated feedback from peer reviews. They note the importance of understanding "the evolving dynamics of technology-mediated learning experiences" and call for further examination of the human-AI relationship in educational contexts.
The authors identify the need to understand how participants perceive the utility of AI reviews in comparison to peer reviews and how emotional responses towards AI influence their reception of feedback. They note the importance of examining the integration of generative AI with peer feedback practices and call for ongoing exploration of how to create optimal feedback ecosystems that combine human and machine intelligence.
Need for understanding how participants distinguish utility of AI-generated feedback from peer reviews
Insights into emotional responses toward AI and their influence on feedback reception
Further exploration of the evolving dynamics of technology-mediated learning experiences
Investigation of how to optimize the integration of peer and generative AI feedback

Extracted from: pdfAgreement 59%

Explore related topics

AI Peer Review Peer Review Mixed Methods