AI evidence extraction

Taking a Pulse on How Generative AI is Reshaping the Software Engineering Research Landscape

Bianca Trinkenreich, Fabio Calefato, Kelly Blincoe, Viggo Tellefsen Wivestad, Antonio Pedro Santos Alves, Júlia Condé Araújo et al. · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Large-scale online survey with mixed methods analysis.

Sample

N = 457, 7 groups

Primary method

Descriptive statistics including absolute counts, proportions, percentages, and distributions. Top-2-Box and Bottom-2-Box scoring for Likert-scale items. Qualitative analysis combining deductive thematic analysis with inductive open coding. Item-level deletion for missing values with no imputation. Segmented analysis by GenAI usage groups. For qualitative coding: consensus-based resolution of disagreements through negotiated agreement across multiple coders.

Main result

The study found that "most of the researchers who use GenAI for research perceive its impact as increasingly pronounced in the near to mid-term future. Specifically, 58% of respondents reported that GenAI has already had a substantial impact (Top-2-Box: a lot or a great deal), rising to 79% for the next year and peaking at 85% for the next five years." Additionally, "GenAI usage is most concentrated in data strategies, with data mining studies showing the highest adoption across all pipeline stages" and "GenAI usage is highest in writing and dissemination, followed by early-stage activities such as goals and design."

Reports effect sizes.

Research paradigm

Mixed methods (predominantly qualitative interpretivism with descriptive quantitative analysis)

Author conclusions

The authors conclude: "Our findings show that GenAI adoption is already widespread, with nearly three-quarters of respondents reporting its use for research. Yet this adoption is uneven across research stages. It usage is concentrated on writing support, summarization, and coding, while research design and data collection see much lower adoption of GenAI." They further state: "Overall, researchers tend to delegate routine and repetitive work to GenAI while retaining control over tasks that require methodological judgment." Finally, they assert: "Looking ahead, we see three priorities. First, the SE community would benefit from shared, evolving guidelines for GenAI use across the research pipeline, grounded in transparency, verification, and accountability. Second, graduate training programs should be designed to ensure that students develop strong research skills alongside GenAI literacy, not as a substitute for it. Third, longitudinal studies are needed to track how the tensions identified in this paper play out as GenAI capabilities evolve and community norms are developed."

Risk of bias

Selection bias: Purposive sampling of authors from top SE venues may not represent all SE researchers (e.g., early-career researchers not yet publishing at top venues, industry practitioners); Respondent bias: Survey respondents may be more engaged with or concerned about GenAI than non-respondents; Single-coder initial bias: Although mitigated by multi-author review, initial coding by one author may have influenced codebook framing; Geographic bias: Sample is largely based in Europe (47%), North America (27%), with underrepresentation from other regions; Gender bias: Predominantly male sample (72%), with minimal representation of non-binary or other genders; Temporal bias: Data collected June-September 2025 during rapid GenAI evolution; findings may be time-sensitive; Selection bias: Purposive sampling limited to published authors in top SE venues may exclude researchers who do not publish or publish in less prominent venues; Response bias: Self-selected respondents may have stronger opinions about GenAI than non-respondents; Single initial coder for qualitative analysis may have influenced codebook framing; Geographic bias: Sample largely based in Europe (47%), North America (27%), with underrepresentation of other regions; Gender bias: Sample predominantly male (72%); Career stage bias: Sample skewed toward early-career researchers (42%); Selection bias: Purposive sampling limited to authors of papers from top SE venues only, which may skew toward more established researchers; Self-selection bias: Survey respondents may be more engaged with GenAI adoption than non-respondents; Social desirability bias: Respondents may over-report or under-report GenAI use depending on perceived community norms; Single initial coder bias: Although reviewed by multiple coders, the initial framing of qualitative codebooks by a single author may have influenced findings; Temporal bias: Survey conducted in 2025; findings may not reflect earlier or later adoption patterns; Geographic bias: Sample is predominantly Europe (47%) and North America (27%); Gender bias: Sample is 72% male

Limitations

The authors acknowledge several limitations: "A potential threat concerns whether the survey instrument adequately captures the construct of GenAI use in research
One respondent noted in the final thoughts question that the questionnaire 'did not transport [the] distinction well' between using GenAI to assist the research process (e.g., writing, brainstorming) and using it as a research tool (e.g., generating test cases or analyzing data)." Additionally, "having a single initial coder may have influenced the framing of each codebook" and "many of our findings, particularly those related to productivity, trust, risk perception, and governance, likely reflect dynamics common across academic disciplines rather than being unique to SE."

Open questions raised

Need for SE-specific vs. discipline-general understanding of GenAI impacts on research
Generalization of findings beyond SE to other academic disciplines
Need for shared experimentation evaluating GenAI's strengths and limitations for various research tasks
Development of specialized GenAI tools (e.g., with SIGSOFT empirical standards knowledge) for research design and data collection phases
Longitudinal studies tracking how researchers' practices and perceptions change as GenAI capabilities evolve
Integration of GenAI literacy into research training programs

Data: Anonymized response dataset available on Figshare at https://figshare.com/s/12b873956384863a7c06; Anonymized response dataset available on Figshare: https://figshare.com/s/12b873956384863a7c06; Anonymized response dataset available on Figshare (https://figshare.com/s/12b873956384863a7c06)Code: No code repositories mentioned; None explicitly mentioned; Not mentionedExtracted from: pdfAgreement 46%

Explore related topics

Responsible AI Use Multi-Domain Survey