AI evidence extraction

Human-AI collaboration patterns in AI-assisted academic writing

Andy Nguyen, Yvonne Hong, Belle Dang, Xiaoshan Huang · Studies in Higher Education · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

3/4

Quality (LMQS)

Evidence

301

Citations

137.94

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1080/03075079.2024.2323593

Methodology & findings

Study design

Experimental study with screen recording data analysis.

Sample

N = 10, 4 groups

Primary method

Shapiro-Wilk test for normality verification (W = 0.91534, p > 0.05); Independent samples t-test to compare mean performance between two clusters; Hidden Markov Model (HMM) with Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) for optimal state selection; Agglomerative Hierarchical Clustering (AHC) using Python scikit-learn library with Silhouette Coefficient; Process mining using Fuzzy Miner approach with Fluxicon Disco software; Quantitative content analysis with constant comparison coding; Hierarchical sequence clustering

Main result

Doctoral students engaging in iterative, highly interactive processes with the GAI-powered assisting tool generally achieve better performance in the writing task. "Findings indicate that doctoral students engaging in iterative, highly interactive processes with the GAI-powered assisting tool generally achieve better performance in the writing task. In contrast, those who use GAI merely as a supplementary information source, maintaining a linear writing approach, tend to get lower writing performance." Specifically, Type 1 (Structured Adaptivity) students demonstrated higher average performance scores (M = 79.75, SD = 20.64) compared to Type 2 (Unstructured Streamline) students (M = 54.75, SD = 10.77).

Reports effect sizes.

Research paradigm

Mixed-methods (quantitative learning analytics with qualitative content analysis)

Author conclusions

"This study points to the need for further investigations into human-AI collaboration in learning in higher education, with implications for tailored educational strategies and solutions." The authors further conclude that "the distinct differences observed in the pathways between the two identified types of writing tactics in human and AI collaboration highlight the crucial importance of purposeful engagement with AI tools in academic writing. High-performing students effectively utilise the GAI-powered tool's functionalities to enrich their writing processes, whereas low-performing students appear to either not make full use of the tool or view it only as an additional resource."

Risk of bias

Small sample size (N=10) with limited generalizability; Sampling bias due to narrow participant population (doctoral students only); Selection bias: participants self-selected for writing task completion; Variability in student interest in writing task and familiarity with subject matter; Single assessment by university lecturer may introduce rater bias; Hawthorne effect: awareness of being recorded may alter behavior; Small sample size (N=10) limiting generalizability and statistical power; Sampling bias risk due to narrow participant population (doctoral students only); Selection bias: only included doctoral students from two universities in Finland and New Zealand; Potential confounding variables: differences in interest in writing task, familiarity with subject matter, and prior AI experience not fully controlled; Single rater/assessor for writing performance (though selection of university lecturer was intended to minimize bias); Hawthorne effect: participants aware of screen recording may have altered behavior; Small sample size (N=10) limits statistical power and increases sampling bias risk; Limited participant diversity restricted to doctoral students in Information Systems or Learning and Educational Technology programs; Self-selection bias: participants were enrolled in specific doctoral programs; Task-specific bias: single writing task on AI in education may not generalize to other writing contexts; Potential confounders: differences in interest in writing task and familiarity with subject matter not controlled; Single assessor for writing performance evaluation (though strategically chosen to minimize bias)

Limitations

The authors state that "the small cohort size not only limits the statistical power of the study but also heightens the risk of sampling bias
This, in turn, could potentially restrict the diversity and representativeness of the sample, thereby impacting the breadth and depth of the insights gleaned from the study." Additionally, "the limited sample size may constrain the extent to which we can confidently extrapolate our findings to the broader population of doctoral students." Furthermore, "the focus on efficiency gains in text production, while significant, only scratches the surface of the broader implications of generative AI technologies on learning processes."

Open questions raised

In-depth investigations into how human writers interact with AI-driven writing assistants
Cognitive and metacognitive factors influencing how students work with AI in academic writing
Methods like eye-tracking or verbal reports for more detailed picture of students' decision-making
Ethical issues including data privacy and potential AI bias
Transformational effects of generative AI on learning paradigms
Innovative assessment tools for evaluating student learning outcomes in context of AI-assisted writing

Data: The data are not publicly available due to privacy or ethical restrictions; data available on request from corresponding author (Andy.Nguyen@oulu.fi).; "The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions."; The data are available on request from the corresponding author (Andy Nguyen, Andy.Nguyen@oulu.fi) but are not publicly available due to privacy or ethical restrictionsCode: Not mentioned in the paper.; Not mentioned in the paperExtracted from: pdfAgreement 52%

Explore related topics

AI Manuscript Writing Human-AI Collaboration Empirical Study