AI evidence extraction

A dancing bear, a colleague, or a sharpened toolbox? The cautious adoption of generative AI technologies in digital humanities research

Rongqian Ma, Meredith Dedema · arXiv (Cornell University) · 2024

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.48550/arxiv.2404.12458

Methodology & findings

Study design

Sequential mixed-methods design combining an online survey screener (n=76 completed responses from DH scholars) with semi-structured interviews (n=15 interviews lasting 45-60 minutes each, audio-recorded and transcribed).

Sample

N = 76, 11 groups

Primary method

Grounded theory approach combined with deductive coding. Interview transcripts analyzed in NVivo with open coding round followed by axial coding. Two authors independently annotated assigned portions using revised codebook. Descriptive statistics from survey responses (percentages reported for survey items).

Main result

The results reveal that DH research communities hold divided opinions and differing imaginaries about the role of GenAI in DH scholarship. While scholars acknowledge the benefits of GenAI in enhancing research efficiency and enabling reskilling, "many remain concerned about its potential to disrupt their intellectual identities." The study found that "DH scholars adopt GenAI to streamline routine tasks and enhance productivity," with "62% of participants reported that GenAI tools 'make coding easier and faster,'" and that "GenAI is not producing a disruptive paradigm shift in the Kuhnian sense but is gradually reconfiguring the actor networks that constitute DH."

Reports effect sizes.

Research paradigm

Interpretivist/Constructivist with mixed-methods empirical grounding

Author conclusions

The authors conclude that "GenAI is not producing a disruptive paradigm shift in the Kuhnian sense but is gradually reconfiguring the actor networks that constitute DH." They state that "GenAI is being enroled as a new kind of actor—sometimes as a tool, sometimes as a collaborator—and is increasingly integrated into research workflows that shapes not only outputs, but also intentions, methods, and epistemic norms." Furthermore, "whether GenAI will stabilize as a core actor in the DH knowledge ecosystem will depend not only on technological capacity, but also on the relational, material, and institutional work of scholars who are reimagining what it means to produce knowledge in the emerging AI-driven information age."

Risk of bias

Selection bias: Participants self-selected into the survey; recruitment through DH/information-science listservs may reach more engaged/tech-forward scholars; Sampling bias: 76 completed survey responses acknowledged as relatively small sample from broader DH community; Survivorship bias: Study captures only current DH scholars; excludes those who left the field; Temporal bias: Data collected in early 2024; rapid GenAI evolution may date findings quickly; Geographic representation bias: 59 DH scholars from diverse locations, but distribution unclear; 14 from U.S., 8 from Europe, 5 from Canada, others underrepresented; Selection bias: Participants self-selected through DH and information-science listservs and authors' personal social channels, potentially skewing toward more engaged scholars; Sampling bias: Small sample size (76 survey responses, 15 interviews) may not represent broader DH community; Temporal bias: Data collected early in GenAI adoption (February-August 2024) may not reflect evolved practices; Geographic clustering: Sample skewed toward U.S. and Europe (22 of 38 locations specified); Institutional bias: Recruitment through professional networks and institutional channels; Selection bias: Recruitment through specialized listservs (DHSI, ADHO) and authors' personal social channels may oversample engaged DH scholars; Sampling bias: Small sample size (76 completed survey responses, 37 GenAI users, 15 interview participants) with unclear representativeness of broader DH community; Attrition: 75 of 151 survey responses were incomplete; reasons not detailed; Self-selection bias: Interview participants self-selected from those expressing interest in follow-up; Temporal bias: Data collected in early 2024 during rapid GenAI evolution; practices may have changed; Geographic bias: Majority of participants from North America and Europe (14 US, 8 Europe, 5 Canada, others); Academic rank bias: Mixed sample includes professors to graduate students; unequal representation across ranks; Early adopter bias: Survey specifically recruited GenAI users, creating selection toward adopters

Limitations

The authors acknowledge several limitations: "First, the relatively small sample size may introduce sampling bias and limit the generalizability of our findings to the broader DH community." Additionally, "ANT's strong focus on localized interactions can obscure the influence of broader, macro-level forces
Regulatory frameworks, legal systems, and economic or cultural structures often shape actor-networks in ways that are difficult to capture through micro-level analysis alone." The authors also note that "because most of our data were collected in 2024, scholars' practices and perceptions may have shifted alongside the rapid evolution of GenAI tools."

Open questions raised

Lack of systematic empirical research examining how emerging GenAI tools are perceived, used, and evaluated in DH research
Limited understanding of how broader institutional contexts (funding mechanisms, disciplinary boundaries) shape GenAI adoption
Need for longitudinal research to track how DH actor networks and scholars' engagement with GenAI evolve over time
Need to examine how metaphors and discourses around GenAI within DH communities evolve and reflect changing scholarly imaginations of its agency
Insufficient analysis of macro-level forces (regulatory frameworks, legal systems, economic structures) shaping adoption
Need to incorporate frameworks beyond ANT (e.g., STIN, communities of practice, information practices) to understand long-term institutionalization

Data: De-identified survey responses: Ma, R., & Dedema, M. (2025). De-identified survey responses for Generative AI in DH Research [Data set]. Zenodo. https://doi.org/10.5281/zenodo.15881361; De-identified survey responses for Generative AI in DH ResearchExtracted from: pdfAgreement 58%

Explore related topics

Responsible AI Use Human-AI Collaboration Mixed Methods