AI evidence extraction

CIRED.digital project final report

Minh Ha-Duong · HAL (Le Centre pour la Communication Scientifique Directe) · 2025

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

Methodology & findings

Study design

Mixed-methods case study combining: (1) technical implementation and deployment of a RAG system over three sequential phases (Development, Integration/Testing, Deployment); (2) observational usage analysis of 259 sessions generating 1,849 events over 96-day beta period; (3) cost analysis using three complementary estimation methodologies (commit-based, component-based, lines-of-code); (4) environmental impact assessment through component-based carbon accounting; (5) qualitative query content analysis using rule-based classification and keyword extraction; (6) user journey mapping through event transition analysis..

Primary method

Design science research with iterative development and user-centered refinement. Three sequential phases: (1) Development phase emphasizing rapid prototyping; (2) Integration and Testing phase with external user feedback; (3) Deployment and Evaluation phase with progressive rollout (closed beta → open beta) and continuous monitoring. Design involved phased deployment strategy, interface design evolution, and modular architecture enabling independent component development.

Main result

The project demonstrates that institutional RAG deployment remains financially accessible to research laboratories, with "production-ready conversational access to scientific publications can be achieved for €40,000-60,000 in development costs and €5,000-10,000 in annual operational costs." The system successfully processed 290 queries across 259 sessions over 96 days, with "generated answers contain 6.1 citations on average (interquartile range 3-8 citations), referring to 2.8 distinct publications (interquartile range 1-4 publications)." Usage analysis shows that "topic synthesis queries (135 instances, 71.8%) dominated the dataset, encompassing broad overview questions... This pattern indicates users primarily sought accessible entry points to CIRED research rather than detailed technical documentation."

Research paradigm

pragmatist/design science

Author conclusions

The authors conclude that "CIRED.digital validates a model of institutional AI systems that serve research missions while remaining community-controlled, transparent, and sustainable in cost and impact" and that "evidence supports institutional RAG deployment as technically feasible, environmentally sustainable at modest scale, and valuable for research dissemination when implemented with attention to privacy, transparency, and appropriate governance." They further state that "institutions need not rely on commercial platforms for AI-mediated access to institutional knowledge: locally controlled systems are feasible, preserve independence, protect usage data, and can contribute to shared research infrastructure." The project demonstrates that "production-ready conversational access to scientific publications can be achieved for €40,000-60,000 in development costs and €5,000-10,000 in annual operational costs—figures well within reach of most research institutions."

Risk of bias

Selection bias: 259 sessions represent self-selected early adopters and CIRED network users rather than representative population; Measurement bias: 59% bounce rate may reflect bots rather than human users; automated Claude-based query classification introduces misclassification risk; Attrition: Low feedback collection (18% feedback rate) and sparse user profile capture (4 events); Sampling bias: Heavy skew toward French and English users with minimal Arabic usage despite multilingual design; Instrumentation bias: System not fully reliable in capturing every interaction; article view events missing for some response→article transitions; Selection bias: Early adopters and CIRED network users overrepresented; 25% of sessions from bots (Google bot); Self-selection bias: Engaged users (23%) fundamentally differ from bounced visitors (59%); Sampling bias: Limited to HAL-CIRED collection (~1,238 documents), incomplete historical record; Measurement bias: Machine-based query classification (Claude AI) introduces potential misclassification compared to manual expert coding; Instrumentation bias: System not fully reliable in capturing every interaction (acknowledged missing article view events); Attrition bias: High bounce rate (59%) limits representativeness of engaged user sample; Institutional bias: CIRED researchers likely overrepresented in CIRED network user segment (18% of sessions); Selection bias: Users self-selected during beta deployment; 59% bounce rate suggests non-representative sample; Attrition: Usage declined sharply after July 2025 public launch, limiting sustained usage analysis; Sampling bias: 'These queries primarily reflect early adopters, invited testers, and CIRED network users rather than representative public usage'; Measurement bias: Machine-based query classification via Claude introduces potential misclassification; Information bias: Missing logging events for article views and incomplete user profile capture (only 4 userProfile events recorded); Geographic bias: Hetzner hosting in Helsinki may not represent typical European infrastructure costs; Language bias: Corpus predominantly French/English with minimal Arabic coverage (3.2%); Institutional bias: 40% of traffic from CIRED network users, non-representative of broader user base

Limitations

The authors acknowledge that "The beta-phase dataset is limited in size and composition and does not support formal statistical inference
The analyses below therefore provide directional and design-relevant insights, not population-level estimates." Additionally, "Machine-based analysis introduces potential misclassification compared to manual expert coding." The corpus is incomplete: "While HAL provides the primary corpus source, it captures an incomplete record of CIRED's 50-year research production." Technical limitations include: "Approximately 5-10% of PDFs required OCR fallback due to scanned-only formats" and "the system is not fully reliable in capturing every interaction" regarding logging events
User study limitations: "the absence of systematic latency logging limited retrospective performance assessment" and "We cannot access detailed time logs from all six contributors, making precise effort allocation impossible."

Open questions raised

Multi-turn conversational context: 30% of user feedback cited lack of conversational memory; follow-up question support requires 2-3 weeks development
Structured output generation for tabular data requests (author bibliographies, model comparisons, country-level statistics)
Enhanced publication discovery integration with HAL metadata and author-topic linking
Advanced retrieval techniques (semantic chunking, SPLADE, ColBERT) for improved relevance
Domain-specific embeddings and fine-tuned smaller models for CIRED-specific answer quality
Formal response quality audits to monitor hallucination rates and citation accuracy

Data: Anonymized usage dataset of 290 user queries and aggregated usage statistics prepared for open archival deposit. GitHub repository: https://github.com/CIRED/cired.digital; Anonymized usage datasets prepared for archival deposit (290 user queries, 1,849 events from 259 sessions over 96-day beta period); Analytics scripts available in project codebase; Usage patterns and query logs (with identifiers removed); Anonymized usage dataset of 290 queries and 1,849 interaction events prepared for open deposit; GitHub repository with codebase: https://github.com/CIRED/cired.digital; HAL-CIRED collection (1,238 documents after filtering) accessible via HAL APICode: https://github.com/CIRED/cired.digital (full codebase published under CeCILL-B open-source license); https://github.com/CIRED/cired.digital/issues (open issues and development tracking); GitHub: https://github.com/CIRED/cired.digital (full codebase published under CeCILL-B license); Git repository with 210 commits to main branch documenting development history; https://github.com/CIRED/cired.digital (complete codebase published under CeCILL-B open-source license); GitHub issues tracker at https://github.com/CIRED/cired.digital/issuesExtracted from: pdfAgreement 43%

Explore related topics

RAG for Research Literature Discovery Case Study