AI evidence extraction

The Role of Large Language Model (LLM)- Reasoning Agents in Conducting IS Surveys

Garan Kim, Naveen Kumar, Heshan Sun · Journal of the Association for Information Systems · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

0/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Proposed methodology: development and validation of LLM-Reasoning Agent Survey (LAS) by constructing reasoning agents with demographic, psychological, behavioral, and contextual profiles.

Main result

The paper proposes that "LLM-reasoning agents can effectively represent sample populations for IS research" by "constructing LLM-reasoning agents assigned demographic, psychological, behavioral, and contextual profiles from an existing survey dataset" and evaluating them "across four critical dimensions: construct validity, internal validity, external validity, and reliability." However, this is a research-in-progress paper with no empirical results presented.

Reports effect sizes.

Research paradigm

Positivist/empiricist

Author conclusions

The authors conclude that "LAS offers potential advantages including access to sensitive topics, simulation of hard-to-reach populations, rapid testing of emerging technologies, and cost-effective replication studies—addressing longstanding challenges in IS survey methodology." However, they emphasize this is "research-in-progress," indicating conclusions are provisional pending empirical validation.

Risk of bias

Potential selection bias in choice of LLM model and configuration; Risk of anthropomorphization bias if LLMs are assumed to replicate human cognition; Sampling bias from existing survey dataset used for profile creation; Potential confounding from prompt design and framing effects on LLM responses; Potential selection bias from using existing survey dataset for profile assignment; Possibility of algorithmic bias embedded in LLM reasoning processes; Limited external validity if agent profiles derived from single dataset; Risk of measurement error if LLM responses do not authentically mirror human cognition; Potential algorithmic bias in LLM responses; Selection bias from using profiles from existing survey datasets; Model-specific biases inherent to the LLM implementation; Lack of human subject validation at research-in-progress stage

Open questions raised

The paper identifies longstanding challenges in IS survey methodology including "common method bias, low response rates, poor data quality, limited access to hard-to-reach populations, and challenges capturing rapidly evolving technology perceptions." The authors propose LAS as a potential solution but acknowledge the need for empirical validation.
Persistent limitations in traditional IS surveys including common method bias, low response rates, poor data quality, limited access to hard-to-reach populations
Challenges capturing rapidly evolving technology perceptions
Need for methodologies that can complement traditional survey approaches
The authors identify that survey methods in IS research "face persistent limitations including common method bias, low response rates, poor data quality, limited access to hard-to-reach populations, and challenges capturing rapidly evolving technology perceptions."

Extracted from: pdfAgreement 74%

Explore related topics

Systematic Review Automation Scholarly Infrastructure Experimental