AI evidence extraction

Report from Workshop on Dialogue alongside Artificial Intelligence

BOA (University of Milano-Bicocca) · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

5/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2511.05625

Methodology & findings

Study design

Workshop synthesis methodology: The document was generated through a systematic computational approach synthesizing exclusively human-authored content from workshop participants.

Primary method

The workshop synthesis document does not employ statistical methods. However, cited empirical studies used: randomized controlled trials (mentioned for CDA studies), preliminary evaluations with overlap calculations (ChatGPT coaching study showing >50% agreement with human coaches), design-based research (UK/Mexico videoconference study), qualitative comparative analysis (mixed-reality teacher training), and exploratory research design (PRODIGY dialogue training). The document notes that "advanced statistical studies and different forms of qualitative analysis are used" in the underlying learning science research, including "coding schemas" and studies that "can integrate AI as part of instructional sequences and learning trajectories."

Main result

Key findings from the workshop highlight that "AI offers transformative opportunities to enhance educational dialogue, yet it also poses risks of undermining the authentic human interaction at the heart of learning." The workshop demonstrated that "AI-based analytics and visualization tools can significantly improve teachers' dialogic practices and even boost student learning outcomes," with one study showing "teachers who used the CDA visual feedback showed greater gains in facilitating rich classroom dialogue (and their students achieved higher learning gains) compared to teachers who engaged in traditional video reflection without AI support." Additionally, research revealed that "when pairs of students engaged with a single AI chatbot together, it spurred deeper discussion and reflection than when students interacted one-on-one with the AI," and that "over half of the segments flagged by the AI matched those that expert human coaches would also have selected."

Reports effect sizes.

Research paradigm

Mixed epistemology (interpretivist, constructivist, pragmatist) with reference to learning sciences and socio-cultural theory

Author conclusions

The workshop concludes that "AI works best as a mediating tool within human-centered tools, networks and infrastructures – not as a replacement for teachers or dialogue partners." The authors emphasize: "Ensuring a human-centered, equitable integration of AI is crucial to harness its benefits while preserving the collaborative, inclusive nature of education." They state that "to be welcomed in the classroom, AI systems must behave in ways that are understandable and answerable to the educational community's values." The group consensus is that "AI should augment rather than supplant human instruction," and that the "human-in-the-loop mindset is essentially a safeguard against abdication of educational responsibility to algorithms – it keeps teachers as orchestrators of learning and students as active learners, with AI as an empowering tool under their guidance."

Risk of bias

Selection bias: Workshop participants were self-selected 'leading researchers' from specific institutions, potentially representing a particular viewpoint in AI-in-education; Publication bias: Workshop contributions likely skew toward positive or promising results rather than null or negative findings; Reporting bias: Empirical studies cited are summarized secondhand through participant presentations rather than original peer-reviewed publications; Confirmation bias: The workshop's framing focused on intersection of dialogue and AI, potentially attracting participants with positive orientations toward AI in education; Institutional bias: Participants come predominantly from well-resourced universities in developed nations (USA, UK, Norway, Israel, Hong Kong, Germany), potentially limiting cross-cultural representation; Technology adoption bias: Workshop settings may privilege early adopters who are more optimistic about AI integration; Selection bias in workshop participant recruitment (19 leading researchers self-selected); Publication bias in studies presented (participants likely selected positive or interesting findings); Reporting bias (only studies presented at workshop included, not systematic search of literature); Lack of representation from Global South researchers despite claimed international scope; Potential funding bias from institutions of affiliated researchers; Selection bias: Workshop participants were 'leading researchers' from specific institutions, potentially not representative of all AI-in-education perspectives; Publication bias: Only presented research and positive outcomes were included; Institutional bias: Participants from major research universities (Harvard, Cambridge, Oslo, Hebrew University, etc.); Temporal bias: Most empirical studies cited are preliminary or early-stage; Language bias: English-medium discussion may exclude non-English scholarship; AI tool bias: Several participants presented tools they developed, potential conflict of interest

Limitations

The document acknowledges several methodological limitations in the underlying studies: (1) Regarding AI coaching for teachers, "the AI sometimes misses the deeper context – for instance, it might flag a moment as suboptimal based purely on transcript text, whereas a human coach knows from classroom context that the teacher's move was appropriate." (2) For mixed-reality teacher training, critics argue "pedagogical dialogue is inherently a human relational practice that might not transfer from a simulation to a real classroom," though the authors note "early findings suggest that while simulations cannot replicate all nuances of human interaction, they do help teachers gain confidence." (3) Regarding literary analysis with AI, researchers found that "students who relied heavily on AI formulations sometimes mirrored the AI's language in their essays, leading to a certain uniformity and a missed opportunity to develop their own voice" and observed "a performance gap widening – higher-achieving students used AI outputs as a springboard (often critically), while some lower-achieving students tended to copy or lightly rephrase the AI's words, reinforcing superficial learning." (4) The document notes that studies examining AI chatbot use found "students who were simply given a chatbot often asked trivial questions or got stuck in unproductive exchanges."

Open questions raised

Longitudinal Impact Studies: Long-term studies on how sustained AI use affects students' dialogic skills and dispositions over time (e.g., following cohorts through high school to examine persistent effects versus novelty effects)
Cross-Cultural Validation: Studies deploying the same intervention across different countries/cultural settings to test generalizability and identify cultural biases in AI systems
Multimodal Analysis Methods: Techniques to analyze multimodal dialogue (speech, text, gestures, drawings) in technology-rich environments using machine learning for image/video analysis
Ethical Framework Development: Guidelines for responsible AI integration including data privacy standards, intellectual property of AI-generated work, and bias prevention
Theoretical Extensions: Development of quantum theory metaphors in dialogic education, mechanistic philosophy applications for learning processes, and actor-network theory extensions including AI actors
Technical Innovations: Improved explainable AI for education, multimodal AI integration handling multiple communication modes simultaneously, and domain-specific AI models tailored to educational contexts

Data: No datasets are explicitly stated as available. The report synthesizes findings from multiple researchers' projects but does not provide open-access datasets.Code: No code repositories are mentioned in the document.Extracted from: pdfAgreement 54%

Explore related topics

AI Literacy Training Doctoral Training Position Paper