AI evidence extraction

The AIR framework for research transparency: a critical analysis of stage-specific AI disclosure in the context of accessibility and research integrity

David Ruttenberg · AI & Society · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

This is an AI analysis. Read the peer-reviewed original at the publisher: https://doi.org/10.1007/s00146-026-03082-x

Methodology & findings

Study design

Critical analysis with embedded inter-rater reliability pilot study.

Sample

N = 15, 1 group

Primary method

Inter-rater reliability analysis using Cohen's kappa (κ) to assess agreement among raters applying the AIR framework to research scenarios.

Main result

The study found that "an inter-rater reliability pilot study (n = 15 raters, nine scenarios, Cohen's κ = 0.72) demonstrates that trained evaluators can apply AIR with substantial agreement while revealing systematic boundary ambiguities." The research identified five major limitations in the AIR framework: false precision in ambiguous practices, inadequate treatment of accessibility-related AI use, stigmatization of legitimate high-band practices, vulnerability to adversarial compliance, and insufficient edge case guidance.

Reports effect sizes.

Research paradigm

Critical interpretivism with epistemological grounding in virtue epistemology

Author conclusions

The authors conclude that "AIR, with the proposed refinements, shows promise as transparency infrastructure, but that implementation requires sustained dialogue among researchers, integrity officers, editors, accessibility advocates, and policymakers to ensure research integrity and inclusion remain interdependent rather than competing aspirations." They propose that "transparency must be understood as a constitutive epistemic virtue rather than a procedural requirement."

Risk of bias

Single-author critical analysis may reflect author perspective bias; Pilot study limited to 15 raters and 9 scenarios, restricting generalizability; Potential bias in scenario construction affecting inter-rater reliability assessment; Limited rater sample (n=15) for reliability assessment; Potential selection bias in pilot study rater recruitment; Limited scenario coverage (nine scenarios) may not represent full complexity of research practices; Single author perspective on framework analysis; Small sample size for inter-rater reliability pilot (n=15 raters); Limited scenario set (only 9 scenarios evaluated); Potential selection bias in rater recruitment (not specified how raters were selected); Single author analysis of theoretical framework may introduce interpretive bias

Limitations

Critical analysis identifies five major limitations: "false precision in ambiguous practices, inadequate treatment of accessibility-related AI use, stigmatization of legitimate high-band practices, vulnerability to adversarial compliance, and insufficient edge case guidance." The paper notes that "implementation requires sustained dialogue among researchers, integrity officers, editors, accessibility advocates, and policymakers to ensure research integrity and inclusion remain interdependent rather than competing aspirations."

Open questions raised

The paper identifies the need for evidence-informed refinements including boundary case designations, a protected A1-Access sub-band for disability accommodations, separation of verification burden from appropriateness judgment, spot-check validation studies, and community-maintained edge case repositories. Future research should address systematic gaps in accessibility considerations and develop community-based approaches to edge case management.
Need for boundary case designations in the AIR framework
Development of protected A1-Access sub-band for disability accommodations
Separation of verification burden from appropriateness judgment
Establishment of spot-check validation studies
Creation of community-maintained edge case repositories

Data: not_statedCode: not_statedExtracted from: pdfAgreement 56%

Explore related topics

Responsible AI Use Research Integrity Mixed Methods