AI evidence extraction

Authors, wordsmiths and ghostwriters: Early career researchers' responses to artificial intelligence

David Clark, David Nicholas, Marzena Świgoń, Abdullah Abrizah, Blanca Rodríguez Bravo, Jorge Revez et al. · Learned Publishing · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.64

FWCI

Top 10%

Impact

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1002/leap.1652

Methodology & findings

Study design

Exploratory qualitative study using semi-structured, in-depth interviews (60-90 minutes each) with a convenience sample of 91 early career researchers (ECRs) from multiple countries (China, Malaysia, Poland, Portugal, Spain, and others).

Sample

N = 91, 12 groups

Primary method

Qualitative thematic analysis combined with semi-quantitative coding. Coded analysis using a 0-9 Likert-type scale for comparative assessment across four dimensions (experience, engagement, utility, representativeness). Full-text retrieval and database analysis of transcripts. Breakdown of data by age, gender, subject discipline, and country. Third-party review of coding to detect obvious issues and provide second opinions based on transcripts alone.

Main result

The study found that "ECRs exhibit mostly limited or moderate levels of experience" with AI, and "regarding levels of engagement, there is a divide with some ECRs exhibiting little or none and others enthusiastically using the technology." Additionally, "what appears a major aspect of Generative AI is the automation of 'wordsmithing' and a prospective 'Ghost Writer in the Machine'," with "wordsmithing" being "key for many ECRs" as they seek to "improve their productivity, presentation and language skills."

Reports effect sizes.

Research paradigm

Qualitative/interpretivist

Author conclusions

The authors conclude: "Looking at demographics, regarding age, the median group (35-37) scored the highest marks, and again not the youngest as might have been expected." They further state that "AI is of interest to everybody regardless of fields" and importantly: "Investment in AI, particularly LLMs, is clearly rapidly advancing the automation of literacy far beyond spell-check, auto-complete and machine translation: creating, a ghost-writer in the machine. This will be something we shall follow-up on in the future study." They also note that "concerns were evident in the context of academic outputs, in particular plagiarism and cheating. The questioning of what constitutes knowledge and distinguishes truth also came up frequently."

Risk of bias

Selection bias: convenience sample of 91 ECRs, not randomly selected; Interviewer bias: diverse international interview team with different perspectives that may have influenced coding; Temporal bias: interviews conducted early in LLM adoption (December 2023-early 2024), may not reflect later adoption patterns; Country-specific bias: Chinese interviews conducted when LLM publicity was at peak but internet restrictions limited access; US and UK data excluded due to low cohort numbers; Responder bias: subjective 0-9 Likert-type scale coding by interviewers comparing across their own interview pools; Language bias: interviews conducted in multiple languages then translated to English, potentially losing nuance; Social desirability bias: potential for respondents to over- or under-report AI usage; Convenience sampling rather than random sampling; Interviewer subjective coding on 0-9 scale may reflect interviewer perspectives; Selection bias: researchers self-selected into interviews; Temporal bias: interviews conducted early 2024 when LLM publicity was at peak in some countries but suppressed in others due to VPN restrictions in China; Country-specific variation: low response rates from UK (n=3) and US (n=4); Language of interviews conducted in multiple languages then translated to English, potential translation effects; Convenience sampling (non-random recruitment); Small sample size (n=91) with unequal country representation (22 from China, 32 from Poland, 10 each from Spain, Malaysia, Portugal, 3 from GB, 4 from US); Subjectivity of Likert-scale coding reflecting interviewer perspectives; Temporal context: interviews conducted early 2024 when LLM awareness was at peak; China interviews in December 2023 before rapid model proliferation; Language translation introducing potential bias; Low response rate to some questions (e.g., representativeness question from China); Cultural factors affecting willingness to comment on colleagues

Limitations

The authors state: "This was a preliminary study attempting to inform and plan for a major study, which would have a larger and more representative cohort of ECRs
Our findings should be treated with caution, more as informed observations, filling a knowledge vacuum." Additionally, "Determining how representative ECRs are in respect to their colleagues is more difficult to determine" and the "coding was not a question to be asked of the ECR, but a comparative assessment by the interviewer" which "as a subjective measure, may reveal as much about the perspectives of our diverse interview team as about the ECRs."

Open questions raised

The authors identify the need for: (1) a larger and more representative cohort of ECRs in a future major international study; (2) follow-up investigation of the 'wordsmithing' and 'ghost-writer in the machine' phenomenon; (3) deeper exploration of concerns about plagiarism and cheating in academic outputs; (4) further investigation of the 'post-truth era' and how to distinguish truth from fiction in AI-generated content; (5) longitudinal tracking of how AI adoption patterns change as the technology matures.
Need for larger and more representative cohort of ECRs in follow-up major international study
Further investigation of 'wordsmithing' and 'ghost-writer in the machine' phenomenon
Distinction between what constitutes knowledge and truth in context of AI-generated content, particularly regarding 'post-truth era' concerns
Job loss versus job gain implications of AI adoption
Ethical implications of AI in scholarly publishing and academic integrity

Extracted from: pdfAgreement 62%

Explore related topics

AI Manuscript Writing Academic Writing Mixed Methods