AI evidence extraction

Structuring Human-AI Productive Interdependence by Strategic Level of Automation Selection for Qualitative Inquiry

Feng Zhou, Jacqueline Meijer-Irons, Ambar Murillo · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Single case study (industry case study at Google's DataSat survey program).

Sample

N = 10000, 3 groups

Primary method

No formal statistical methods employed. The analysis is qualitative, based on retrospective assessment of the three-phase workflow. The authors employed Interdependence Theory as an analytical lens and organized findings around the Level of Automation (LoA) framework.

Main result

The study found that effective AI-assisted qualitative analysis requires strategic structuring of human-AI interdependence through Level of Automation selection. The authors demonstrate that "the researcher's primary role is a collaboration architect who structures the productive interdependence: the rules, the tasks, and how outcomes are linked." The three-phase workflow implemented at Google's DataSat program showed that high-risk interpretive tasks require low automation levels with human validation, while low-risk coding tasks can accommodate higher automation with transparent reasoning mechanisms like CART framework prompting.

Reports effect sizes.

Research paradigm

Interpretivism/Constructivism

Author conclusions

"The effective and responsible use of AI in qualitative analysis requires a paradigm shift. We move beyond the pursuit of automation and instead become 'collaboration architects', designing for productive interdependence." The authors further conclude that "by investing in foundational structures, creating asymmetrical roles, and building in mechanisms for validation, we can create Human-AI partnerships that honor the core tenets of qualitative research. The goal is not an automated analyst, but an AI-assisted interpreter—a partnership that allows us to leverage the power of LLMs for scale while ensuring the final act of meaning-making remains where it belongs: in the hands of the human."

Risk of bias

Implicit design of AI role boundaries may lead naive users to misattribute understanding to AI; Passive AI-to-human feedback loop may fail to flag methodological issues; Researcher confirmation bias in interpreting AI outputs not explicitly addressed; Single case study setting may limit generalizability; Implicit rather than explicit role boundaries could enable user misinterpretation of AI capabilities; Passive AI-to-human feedback mechanisms may mask instrumental quality issues; Researcher positionality not declared (authors are Google employees analyzing internal organizational data); Selection bias: Single organization (Google) may not generalize to other contexts; Observer bias: Authors appear to be insider researchers at Google, potentially biasing their assessment of what 'worked well'; Confirmation bias: Retrospective case study analysis vulnerable to selective interpretation of outcomes; Limited external validation: No independent verification of the effectiveness claims; Context-specific design: Solutions tailored to internal survey analysis may not transfer to other qualitative domains

Limitations

The authors identify several implicit limitations: "This separation was entirely implicit in the design
A naive user could still mistake the AI's confident tone for genuine understanding, a classic pitfall, assuming the AI understood the deep context behind user complaints about a particular feature." Additionally, they note that "The quality of the 'AI Insights' can be brittle
Generic insights can feel unhelpful, breaking the cooperative feeling and tempting the researcher to ignore the AI entirely." The authors also acknowledge that "The AI-to-Human validation loop was passive" and that their framework "proposes a static LoA for each task" rather than adaptive approaches.

Open questions raised

How can a system infer a researcher's trust and use it to propose a shift in the interdependent structure? The authors note: "Our framework proposes a static LoA for each task. A more advanced system would navigate the interdependence structure with adaptiveness."
What are the most effective system-initiated strategies for trust repair when AI violates trust pillars?
How to design systems that explicitly communicate AI role boundaries to prevent user misattribution of understanding?
How can a system infer a researcher's trust and use it to propose a shift in the interdependent structure? The authors note their framework proposes a static LoA for each task but identify need for adaptive systems.
What are the most effective system-initiated strategies for trust repair when AI violates pillars of trust (e.g., hallucinations)?
How can systems design for explicit rather than implicit boundaries of AI capabilities?

Extracted from: pdfAgreement 69%

Explore related topics

Qualitative Coding with AI Human-AI Collaboration Case Study