AI evidence extraction

A design science blueprint for an orchestrated AI assistant in doctoral supervision

Teo Sušnjak, Timothy R. McIntosh, Paul Watters · Technology Pedagogy and Education · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1080/1475939x.2026.2660721

Methodology & findings

Study design

Design science research (DSR) approach comprising two phases: (1) literature grounding through targeted review of doctoral identity formation, task distribution, supervision gaps and LLM capabilities; (2) artefact design and justification using Stakeholder Theory and Academic Integrity Principles as theoretical anchors.

Primary method

Design Science Research (DSR) with design-justification cycles of rigour (literature and theory synthesis) and design (iterative refinement against stated constraints)

Main result

The study found that "the orchestrated assistant's agentic capabilities and MoE routing" can address recurrent supervision pain points, with the most severe issues (mental-health strain, harassment, systemic power asymmetries) being human and structural in nature. The analysis supports "a sequencing principle for institutional strategy: First, use the orchestrated assistant's agentic capabilities and MoE routing to call institutionally approved domain-specific models for pastoral-care screening and triage." The assistant demonstrates highest potential on issues with high mitigability, especially "feedback timeliness (S = 3, mitigation = 3), information overload (S = 2, mitigation = 2) and writing load (S = 2, mitigation = 3)."

Research paradigm

Design Science Research (DSR)

Author conclusions

"The genie is out of the bottle. PhD students are already heavily using and increasingly relying on LLMs and ancillary technologies across all of their research and study tasks." The authors conclude that "no technological breakthroughs are required to build an agentic orchestration layer" and that success should be measured "not by engagement with the tool itself, but by its impact on shifting the needle on PhD completion rates and ultimately meaningful pedagogical outcomes: the quality of scholarly work, progress towards milestones and the enrichment of the human supervisory relationship." Crucially, "the human supervisor remains the irreplaceable navigator. They are the conduits of tacit knowledge, the gatekeepers to the community of practice and the mentors of the whole person."

Risk of bias

Literature selection bias: review scope limited to recent sources on PhD supervision and LLM capabilities; Stakeholder representation bias: model excludes external examiners, funders, and industry partners; Disciplinary generalization bias: proposal generic across fields but supervision practices vary significantly by discipline; Temporal bias: rapid evolution of AI models and tools may outpace analysis; Theoretically-driven bias: framework anchored in specific theories (Stakeholder Theory, Academic Integrity Principles, Bloom's Taxonomy) which constrain design space; No empirical data collection - susceptible to confirmation bias in literature interpretation; Limited stakeholder perspectives (three primary stakeholders only; excludes external examiners, funders, industry partners); Generic design across disciplines may not capture discipline-specific supervision practices and risks; Authors acknowledge potential for modeling bias in LLM components (bias and disparate impact noted as risk category); Selection bias in literature review methodology not explicitly described as systematic; No empirical data collection limits validation of design claims; Generic cross-disciplinary design may not capture discipline-specific supervision practices; Literature review scope not explicitly described as systematic search; Ex ante evaluation without user validation

Limitations

This is an "ex ante, literature-grounded design rather than an empirical evaluation." The authors acknowledge that "models and tools change fast and may outpace parts of the analysis." Additionally, "we centre three primary stakeholders and do not model external examiners, funders or industry partners
Our proposal is generic across disciplines, and because supervision practices and AI uptake differ by field, its utility will vary across subjects."

Open questions raised

Lack of empirical field trials across disciplines
Insufficient measurement of learning gains and transfer effects
Need for longitudinal studies tracking supervision dynamics and completion rates
Absence of comparative benchmarking against human-only baselines
Limited governance research on disclosure, privacy, and review load
Under-exploration of LLM role as co-piloting tool in PhD supervision

Code: OpenClaw (https://openclaw.ai/); NanoBot (https://github.com/nanobot-ai/nanobot); https://openclaw.ai/ (OpenClaw agentic framework referenced as example); https://github.com/nanobot-ai/nanobot (NanoBot framework referenced as example)Extracted from: pdfAgreement 69%

Explore related topics

AI Literacy Training Doctoral Training Design Science