AI evidence extraction

The Agentic Researcher: A Practical Guide to AI-Assisted Research in Mathematics and Machine Learning

Max Zimmer, Nico Pelleriti, Christophe Roux, Sebastian Pokutta · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/2

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Mixed-methods case study research with framework development.

Main result

The paper establishes a practical taxonomy of five AI integration levels in research and presents an open-source agentic framework with ten commandments that guide AI agents through autonomous research workflows. Key findings include: "Through repeated interactions of this kind, new results emerge in a process of Human-AI co-creation." The framework demonstrates effectiveness across six diverse case studies spanning machine learning and mathematics, showing that "specific parts of the research workflow can be significantly accelerated when a researcher directs an AI agent in a structured way." For instance, in the LLM optimizer case study, "the best configuration achieved a ∼5% improvement in validation perplexity over Muon (and ∼8% over AdamW) at the same 2N memory budget as AdamW."

Research paradigm

pragmatist (human-AI co-creation in structured research workflows)

Author conclusions

The authors conclude that mathematical and ML research should be approached as human-AI co-creation rather than full automation: "We believe that mathematical research is not a fully automatable task, and we will not speculate on whether this will change in the future. What we do claim is that mathematicians and researchers in general should take an active role in this partial transformation of the field and, echoing Avigad (2026), should own the technology." They emphasize researcher agency: "the researcher still faces a non-trivial amount of work both throughout and toward the end of a project: reviewing intermediate results and providing steering, verifying correctness, deciding what results merit publication, and confirming originality as well as adding context and interpretation. However, instead of conducting the entire research process alone, the researcher now externalizes parts of the work to a capable research associate who delivers a structured, well-documented report."

Risk of bias

Selection bias in case studies: all six cases are chosen to demonstrate successful outcomes with the framework; Potential publication bias: negative results or failed agentic sessions may be underrepresented; Author bias: the framework design reflects the preferences and past failure modes of the four authors; Model-specific bias: findings depend on frontier LLM capabilities (Claude, GPT, Gemini) and may not generalize to other models; Domain bias: case studies concentrated in machine learning and mathematical research; generalization to other domains unclear; Operator bias: researcher skill in crafting initial instructions and steering the agent affects outcomes; Selection bias in case study choice: six case studies selected may not represent full diversity of research domains; Confirmation bias: case studies presented show successes; failure modes are discussed but not quantified proportionally; Researcher positionality: authors developed the framework and selected their own case studies, creating potential conflict of interest; Context window compaction bias: agent may forget or revisit earlier findings unpredictably; Hallucination bias: agents may fabricate citations or mathematical claims despite commandments against this; Selection bias in case study choices (both successful and failure cases documented, but limited to 6 case studies); No systematic comparison to human researchers conducting same tasks; Context window limitations may cause loss of important experimental details; Cost barriers may limit reproducibility for resource-constrained researchers; Agent hallucination in citation generation (despite commandment III)

Limitations

The authors identify several fundamental limitations: "A fundamental limitation of our framework, shared with other agentic systems, is result verification
Natural-language proofs remain difficult to verify and require manual inspection
While code is usually easier to check, subtle implementation errors can still invalidate conclusions." Additionally: "Long experimental sessions with many runs and large outputs can exceed a model's context window and trigger compaction
Because compaction is inherently lossy, the agent may forget details from earlier in the session, revisit failed approaches, or miss important observations." The framework is also limited by cost: "Long autonomous sessions with frontier models can incur nontrivial API costs." Furthermore, the authors state: "The agent may pursue an unproductive direction for too long, especially when the research plan lacks sufficient detail
Verification is only partially solved: while we provide strategies for symbolic and numerical verification of mathematical claims and implementations, a high (to full) degree of certainty requires the researcher to perform a rigorous review of the work."

Open questions raised

Extension to other domains (physics, chemistry, economics, social sciences) beyond mathematics and ML, requiring domain-specific tools and evaluation protocols
More robust memory mechanisms for long research sessions to maintain continuity across experiments and avoid revisiting failed approaches
Multi-user collaboration support, extending the framework from single-user single-agent to multi-user and/or multi-agent settings with coordination and conflict resolution
Context window management for long-running sessions, with practical limitations acknowledged despite mitigations
Systematic benchmarking of agentic research workflows, as noted in the literature survey
Practical question of how researchers should integrate agentic systems into everyday research workflows: "Much less attention has been given to the complementary practical question of how researchers should integrate such systems into everyday research."

Data: C4 (Raffel et al., 2020) - used for calibration data in LLM pruning and quantization case studies; WikiText (Merity et al., 2016) - used for quality evaluation (perplexity on test set) in pruning/quantization studies; FineWeb - used as pretraining data in LLM optimizer experiments (10,000 iterations on 124M-parameter Llama); First Proof (Abouzaid et al., 2026) - benchmark of research-level mathematics questions; FineWeb dataset (for LLM pretraining benchmarks in case study 4.1); C4 dataset (for calibration in LLM pruning and quantization case studies 4.2 and 4.3); WikiText test set (for quality measurement in case studies 4.2 and 4.3); First Proof benchmark (author-curated unpublished research-level mathematics questions mentioned as evaluation source for Aletheia); K7 power network data (case study 4.6, adapted from Lindberg et al. 2020); FineWeb (used in LLM pretraining case study); C4 (used in LLM pruning and quantization case studies); WikiText (evaluation metric for pruning and quantization); First Proof benchmark (referenced, authors' curated problem set)Code: GitHub repository containing the agentic research framework (foot_1 reference in text, full URL not provided in document but referenced as cloneable with setup in README.md); Boscia solver (https://github.com/IST-DASLab/gptq) - mixed-integer optimization branch-and-bound solver with dual tightening implementation; github.com/ZIB-IOL/The-Agentic-Researcher (main project repository mentioned in Section 3.1 and Level 4 setup instructions); Boscia solver repository (case study 4.5 integration); Semenov et al. (2025) standardized LLM pretraining benchmark codebase (case study 4.1); github.com/ZIB-IOL/The-Agentic-Researcher (mentioned as project repository); Semenov et al. (2025) Muon codebase (referenced for pretraining experiments); Boscia solver (integrated implementation in case study 4.5)Extracted from: pdfAgreement 33%

Explore related topics

Responsible AI Use Human-AI Collaboration Conceptual