AI evidence extraction

Automated Text Generation and Summarization for Academic Writing

Fernando Benites, Alice Delorme Benites, Chris M. Anson · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

5.47

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1007/978-3-031-36033-6_18

Methodology & findings

Study design

Narrative literature review of computational linguistics and NLP technologies; critical analysis of text generation and summarization systems; examination of evaluation metrics (perplexity, BLEU, ROUGE, METEOR); historical tracing of AI language processing from early rule-based systems (Talespin) to modern transformer architectures (BERT, GPT-3); case study analysis of ChatGPT output..

Main result

The paper finds that "Automated text generation has undergone significant advances in the last few years and is likely to redefine human–machine writing interaction in the near future." The authors demonstrate that while neural network-based systems like GPT-3 and transformers can generate fluent and contextually appropriate text, "the content suggested by automatic systems is not justified by semantic or extra-linguistic criteria," and these systems "do not understand the words that they are processing." Furthermore, the research shows that automatic text generators can function in two primary modes: generating first drafts from procedural input, or post-editing raw drafts to enhance fluency and style.

Research paradigm

interpretivist/critical discourse analysis

Author conclusions

The authors conclude that "automatic text generators can develop into widely-used writing assistance devices, where humans still perform various parts of the writing process. However, it is difficult to foresee precisely how the use of such automated solutions will change the traditional theoretical stages of writing (e.g., planning, prewriting, drafting, and revising)." They further argue that "notions that proved useful to explain, analyze, and even teach academic writing, i.e. Swales' (1990) CARS model of rhetorical moves, will need to be re-examined in the light of human–machine-interaction." Importantly, they stress that "a fruitful collaboration with the machine in order to produce good academic texts requires that the user knows how to make the best of the possibilities it offers and remain in control of the writing process."

Risk of bias

As a narrative review, bias risks include: selection bias in literature chosen for discussion; potential confirmation bias in presenting technology limitations; reliance on self-selected exemplars rather than systematic literature survey; subjective interpretation of AI capabilities and risks.; Industry claims tend to be more enthusiastic and less rational than justified; Subjectivity in reference text selection for evaluation; Human evaluator disagreement on text quality assessment; Potential replication of problematic assumptions from training data (e.g., gender and occupational stereotypes)

Limitations

The authors note that "it is difficult to foresee precisely how the use of such automated solutions will change the traditional theoretical stages of writing" and that "at the moment, there is no evidence that the systems take textual or pragmatic constraints into consideration
in other words, information structure, intertextuality, and rhetorical development cannot be expected to be part of an automated writing process." Additionally, they acknowledge that "the question of quality evaluation is not resolved yet" for automatic text generation systems, and automatic metrics like BLEU and ROUGE "do not evaluate whether the meaning of a text is correctly conveyed—they merely check if the right words have been used."

Open questions raised

The authors identify several gaps: (1) how automatic text generation will change traditional stages of writing (planning, prewriting, drafting, revising); (2) how AI-generated content can be accounted for in social constructivist theories of writing; (3) whether automatic systems can address pragmatic constraints like information structure, intertextuality, and rhetorical development; (4) the need for better evaluation metrics that assess semantic correctness rather than just word overlap; (5) how multi-source text summarization for literature reviews can be automated when current technology focuses only on word/sentence extraction.
The authors identify that understanding how automatic text generation will change traditional writing stages (planning, prewriting, drafting, revising) remains unclear. They note that accounting for automated text generation use in social constructivist theories of writing is problematic since automated system suggestions lack semantic and extra-linguistic justification. The authors also identify gaps in how pragmatics, information structure, intertextuality, and rhetorical development can be incorporated into automated writing processes. Additionally, they highlight unresolved questions about quality evaluation metrics and their alignment with human evaluation logic.
How automatic text generation will change the traditional theoretical stages of writing (planning, prewriting, drafting, revising)
How different uses of automatic text generators can be accounted for in social constructivist theories of writing
Whether automatic systems can address pragmatic constraints (information structure, intertextuality, rhetorical development)
How evaluation metrics can assess meaning preservation in abstractive summarization

Extracted from: pdfAgreement 58%

Explore related topics

AI Manuscript Writing Academic Writing Literature Review