AI evidence extraction

The Relic Condition: When Published Scholarship Becomes Material for Its Own Replacement

Deng Lin, Chang-bo Liu · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed-methods case study combining corpus extraction, expert evaluation, and student usability assessment.

Sample

N = 13, 4 groups

Primary method

Descriptive statistics only. The student survey used means, standard deviations, medians, and ranges on 7-point Likert scales. No inferential statistical tests are reported. Panel discussion scores are reported as raw 10-point ratings. Expert evaluation reports use qualitative rubric-based assessment with some numeric scoring (5-point and 10-point scales) but without statistical aggregation or hypothesis testing.

Main result

The study found that "all review and supervision reports in the preserved archive judged the outputs to meet or exceed benchmark" and that "appointment-level recommendations placed both bots at or above Senior Lecturer level in the Australian university system (broadly equivalent to tenured Associate Professor in the US system)." Scholar A achieved panel scores "between 7.9 and 8.9/10" while Scholar B achieved scores "between 8.5 and 8.9/10 under multi-turn debate conditions."

Reports effect sizes.

Research paradigm

Empirical-interpretive: combines experimental testing of AI systems with qualitative expert evaluation and theoretical analysis

Author conclusions

"This paper has reported a small-scale but high-density demonstration that the reasoning systems of established humanities and social science scholars can be distilled from their published corpora alone and deployed as functional academic labour. The resulting scholar-bots were judged benchmark-attaining across review, supervision, lecturing and panel exchange by independent senior academics, and classified at Senior Lecturer level or above in all six appointment-level syntheses." The authors conclude that "the technical threshold for scholarly reasoning capture has already been crossed under ordinary publication conditions, using public corpora, no domain fine-tuning and modest engineering effort" and "The window for protective action is the present. Disclosure requirements, consent frameworks, compensation mechanisms and deployment restrictions are all technically feasible and politically achievable while scholarly reasoning distillation remains a demonstrated capability rather than an entrenched infrastructure."

Risk of bias

Social proximity bias: All 10 student survey participants were personally known to the author; Selection bias: Only two scholars in a single subfield selected; not representative of all humanities/social science scholars; Evaluator familiarity: Expert evaluators may have been aware of the authors' theoretical positions; Lack of blinding: Expert evaluators were not blinded to whether outputs came from distilled systems versus control conditions; Heterogeneous expert evaluation: Different evaluators used different rubrics and formats; Self-authorization bias: The doctoral proposal, peer-review manuscript and panel materials were all authored by the present author; Selection bias: Only two scholars selected from single subfield of humanities and social sciences; Social desirability bias: Student usability cohort was personally known to author; Lack of blinding in expert evaluation; Small sample size for student survey (n=10) limits generalizability; Evaluator heterogeneity in rubric application and report format; Author-created evaluation inputs (doctoral proposal, manuscript, lecture content); Selection bias: Only two scholars selected from critical heritage studies; both had large public corpora making them atypical; Evaluator bias: Three evaluators were senior academics with potential professional interest in AI capabilities; no masking/blinding reported; Social desirability bias: Student participants (n=10) were socially proximate to author, reducing anonymity and potentially inflating ratings; Small sample bias: Student survey (n=10) provides limited statistical power; ceiling effects noted on 7-point scale; Corpus selection bias: Selection of scholars based on legibility, taxonomization and extractability may not represent typical scholars; Incomplete baseline comparison: Baseline trials with general chat interfaces (ChatGPT, Claude, Gemini) noted but not tabulated with same archival fullness; Publication bias: Study not pre-registered; single positive result in narrow domain; Attrition: No follow-up data on whether scholars approved or objected to distillation after disclosure

Limitations

The study has seven main limitations: "First, the two primary scholar corpora were not recorded with identical unit conventions, so corpus scale is similar in ambition but not perfectly symmetrical in accounting
Second, the panel archive remains heterogeneous in format across rounds
Third, expert reports vary in form
Fourth, the student-usability cohort was socially proximate to the author, interacted through a guided local Codex interface, and comprised only 10 participants, which limits statistical power and generalizability
Fifth, the distillation reconstructs public intellectual practice, not the private totality of a scholar's mind
Sixth, the paper deliberately withholds full extraction prompts, skill files and reproducible pipeline artefacts for dual-use reasons, which constrains exact reproducibility

Open questions raised

The authors identify the need for further research into: (1) which scholarly reasoning architectures under which publication conditions are most exposed to analogous capture, (2) cross-disciplinary uniformity of scholarly distillability, (3) prevalence and generality of the phenomenon, (4) governance frameworks for AI-captured reasoning in research, (5) strategic identification of which dimensions of scholarly life must remain human, and (6) rethinking of doctoral education, mentorship, peer evaluation and scholarly publication under extraction conditions.
Generality and prevalence of scholarly distillability across disciplines and scholar types
Cross-disciplinary uniformity of the Relic condition
Which scholarly reasoning architectures and publication conditions are most exposed to capture
Institutional and governance frameworks needed to protect scholarly labor
Long-term effects on knowledge-production ecology if distillation becomes infrastructural

Data: No publicly available datasets mentioned. Student survey data aggregated and reported descriptively without individual-level release. Author notes supplementary information file (S1-S32) accompanies paper but sensitive extraction artefacts and skill-module files withheld.Code: None mentioned. Paper explicitly withholds "full extraction prompts, skill-module files and reproducible pipeline artefacts for dual-use reasons"Extracted from: pdfAgreement 52%

Explore related topics

Responsible AI Use Human-AI Collaboration Case Study