AI evidence extraction

Using AI to write scholarly publications

Mohammad Hosseini, Lisa M. Rasmussen, David B. Resnik · Accountability in Research · 2023

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

10/10

Relevance

0/4

Quality (LMQS)

Evidence

264

Citations

42.65

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1080/08989621.2023.2168535

Methodology & findings

Study design

Philosophical and ethical analysis with illustrative case examples of NLP system outputs (ChatGPT responses to specific prompts); normative argument development; no empirical study or simulation

Main result

NLP systems can "generate coherent and informative text, ranging from a few sentences or paragraphs to an entire essay in response to specific prompts from the user" and demonstrate impressive capabilities, but they are also prone to significant errors. The systems can "Hallucinate," "are Frequency-Biased" and "are often Confident But Wrong," and they "do not (yet) have the type of cognition or perception needed to understand language and its relationship to the external physical, biological, and social world."

Research paradigm

Critical-normative (ethical/philosophical analysis)

Author conclusions

Authors conclude that "any section of a manuscript written by an NLP system should be checked by a domain expert for accuracy, bias, relevance, and reasoning" and that "authors are ultimately responsible for the text generated by NLP systems and must be held accountable for inaccuracies, fallacies, or any other problems in manuscripts." The editors announce they are "planning to adopt a policy on the inclusion of text and ideas generated by such systems in submissions to the Journal" with goals to "ensure transparency and accountability related to use of these systems, while also being practical and straightforward."

Risk of bias

AI/ML systems reflect biases in training data; NLP systems may reproduce or amplify racial, gender, or other biases present in training datasets; Potential for hallucination and confidence in incorrect outputs; AI systems reflect biases in training data (e.g., racial, gender biases); Frequency bias in language models; Selection bias in data used for training NLP systems; Potential for bias reproduction in NLP systems trained on biased data; Author selection of illustrative examples may not represent full range of NLP capabilities/failures; Editorial perspective reflects specific disciplinary concerns in research integrity

Limitations

The authors acknowledge that "NLP systems are likely to continue to make factual and commonsense reasoning mistakes because they do not (yet) have the type of cognition or perception needed to understand language and its relationship to the external physical, biological, and social world." Additionally, "NLP systems can perform well when working with text already created or curated by humans, but can perform (dangerously) poorly when they lack human-generated data related to a topic and try to piece together text from different sources."

Open questions raised

Philosophical questions about whether NLP systems are intelligent and what this means for human intelligence
Issues of originality and author intellectual contribution in the context of NLP-assisted writing
Future exploration of authorship rights for AI systems (though authors note this day has not yet arrived)
Broader implications of AI use in academic integrity across colleges, universities, and K-12 education
Ethics of employing trainers and NLP systems' need for massive human and financial resources
The paper identifies the need for: clearer guidelines on authorship and NLP contribution disclosure; further exploration of ethical issues related to NLP trainers and resource requirements; development of detection systems for NLP-generated content; policies across academic journals; and future consideration of authorship rights for AI systems as their capabilities evolve.

Extracted from: pdfAgreement 70%

Explore related topics

AI Manuscript Writing Academic Writing Position Paper