AI evidence extraction

Democratizing Writing Support with AI: Insights from One Year of Real-World Interactions with an Open-Access Writing Feedback Tool

Babette Bühler, Ivo Bueno, Enkelejda Kasneci · Proceedings of the AAAI Conference on Artificial Intelligence · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

7/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1609/aaai.v40i45.41167

Methodology & findings

Study design

Large-scale observational analysis of voluntary user interactions with an open-access AI writing feedback system.

Sample

N = 23650, 5 groups

Primary method

Clustering approach to identify iterative revision chains; LLM-based multidimensional scoring framework for text quality assessment; correlation analysis (r = .17); significance testing (p < .001).

Main result

The study found that "students who revised their texts after receiving AI feedback demonstrated statistically significant, albeit modest, improvements across both content and language-related dimensions (overall writing quality: ∆ = 0.067, p < .001, r = .17), with the greatest gains observed among initially low-performing writers." Additionally, "revision frequency was positively associated with improvement, particularly in higher-order writing skills."

Reports effect sizes and confidence intervals.

Research paradigm

Empiricist/Positivist

Author conclusions

The authors conclude that "these results demonstrate both the technical feasibility and social potential of deploying generative AI for educational support at scale, while highlighting the need for inclusive infrastructure, accessible design, and targeted outreach to truly democratize educational benefits."

Risk of bias

Selection bias: Higher usage among students in academically oriented schools; Self-selection bias: Voluntary interactions with the open-access system; Potential confounding variables: Educational context and age group differences not fully controlled; Attrition: Unknown dropout rates among the 23,650 interactions; Self-selection bias: Voluntary participation in open-access tool; Confounding variables: Unmeasured individual differences in motivation, prior writing ability, and access to alternative feedback sources; Attrition bias: Potentially different dropout rates across demographic groups; Selection bias: Voluntary interactions only; self-selected users of open-access tool; Socioeconomic/educational bias: Higher usage among academically oriented schools suggests unequal access; Confounding variables: No randomized control group; cannot isolate AI feedback effect from other writing instruction; Attrition: Not specified whether users who stopped engaging are included in analysis; Measurement bias: LLM-based scoring framework may have inherent biases

Limitations

The authors note that "engagement was uneven, with higher usage among students in academically oriented schools," indicating potential selection bias
The paper emphasizes the need for "inclusive infrastructure, accessible design, and targeted outreach to truly democratize educational benefits," suggesting limitations in current reach and equitable access.

Open questions raised

The authors identify the need for inclusive infrastructure, accessible design, and targeted outreach to democratize educational benefits of AI writing feedback tools, particularly to reach underrepresented populations in academically oriented schools.
How to achieve equitable access and usage across different educational contexts and demographic groups
Strategies for inclusive infrastructure and accessible design of AI writing support tools
Understanding barriers to engagement among underrepresented student populations
Long-term effects and sustainability of AI-based writing support systems
The authors identify the need for: (1) inclusive infrastructure and accessible design for AI writing support tools, (2) targeted outreach to underrepresented student populations, and (3) further research on how to extend benefits beyond academically-oriented school settings.

Data: not_statedCode: not_statedExtracted from: pdfAgreement 58%

Explore related topics

AI Manuscript Writing Academic Writing Empirical Study