AI evidence extraction

FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics

Qiran Zou, Hou Hei Lam, Wenhao Zhao, Tingting Chen, Yiming Tang, Samson Yu et al. · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Controlled empirical comparison study: Six AI research agents were evaluated on 18 machine learning research tasks across 10 domains.

Sample

N = 324, 14 groups

Primary method

Pooled Spearman rank correlation analysis over 324 (round, agent, task) runs; p-value significance testing at α=0.05; per-agent mean computation with standard error reporting; pairwise win-rate analysis computed as strict-win fraction against every other agent; GraphCodeBERT embedding-based code similarity analysis

Main result

The study found three main findings: (1) "strategy complexity alone does not guarantee strong performance: a simple greedy hill-climber nearly matches the best-performing tree-search agent, both well above the remaining agents"; (2) "greedy search tends to be more effective when opportunities are dense, while tree-search and evolutionary strategies tend to be more effective when opportunities are sparse"; and (3) "early convergence and directionally focused exploration are significantly associated with final performance, while solution diversity and compute cost are not."

Reports effect sizes.

Research paradigm

Positivist/empiricist

Author conclusions

"We introduced FML-bench, comprising 18 fundamental ML research tasks across 10 domains, and used it to conduct a controlled comparison of six representative AI research agents by separating execution infrastructure from agent strategy and defining twelve process-level behavioral metrics. We find that strategy complexity alone does not guarantee performance; greedy search tends to be more effective when improvement opportunities are dense while broader strategies tend to be more effective when opportunities are sparse; and early convergence speed and directionally focused exploration are significantly associated with final performance, while solution diversity and compute cost are not."

Risk of bias

Selection bias in agent choice: six agents selected may not be fully representative of all agent strategy types; Infrastructure removal bias: stripping auxiliary subsystems (literature retrieval, VLM-based analysis, multi-seed averaging) may differentially impact agents designed to leverage these components; Fixed budget bias: the T=100 step budget may favor certain agent types over others; Task selection bias: 18 tasks across 10 domains may not uniformly represent ML research space; LLM capability confound: all agents use the same underlying LLM (GPT-5.4), which may mask differences in agent-LLM alignment; Selection bias: Only six agents selected; claims that they "cover the major search topologies but are not exhaustive"; Infrastructure removal bias: Uniformly removing native code editors and auxiliary subsystems may underestimate agent performance relative to original configurations; Budget bias: Fixed 100-step budget may differentially affect agents with different computational strategies; LLM selection bias: All agents used GPT-5.4, potentially affecting relative performance across different agents' design assumptions; Task selection bias: 18 tasks may not represent full diversity of ML research domains; Removal of native subsystems may systematically disadvantage agents whose original designs relied on auxiliary components; Limited to 6 agents - may not represent full diversity of emerging strategies; Fixed 100-step budget may favor certain search topologies over others; All runs used single LLM (GPT-5.4) - generalization to other LLMs unclear; Task selection bias: 18 tasks across 10 domains may not represent full ML research landscape

Limitations

"The main limitations of this work are that, to enable controlled comparison, we uniformly removed each agent's native code editor and auxiliary subsystems (Appendix C), which may underestimate each agent's performance under its full original configuration
six agents cover the major search topologies but are not exhaustive
and the fixed step budget may affect relative rankings."

Open questions raised

The benchmark enables future investigation of online strategy switching and adaptive agent design
Broader agent coverage beyond the six evaluated agents
Investigation of how performance rankings may shift under alternative step budgets
Further exploration of the relationship between task improvement opportunity structure and agent strategy effectiveness
Broader agent coverage beyond six agents to ensure exhaustive coverage of search topology space
Investigation of online strategy switching mechanisms

Data: FML-bench benchmark: https://github.com/qrzou/FML-bench (18 tasks across 10 domains: DomainBed-CM, DomainBed-OH, EasyFSL, USB, Lightly, Solo-learn, Cont.-Learn., PyCIL, CausalML, gCastle, ART, OpenOOD, PrivacyMeter, Opacus, AIF360, Fairlearn, Unlearning, PFLlib); FML-bench: https://github.com/qrzou/FML-bench; ColoredMNIST (DomainBed-CM task); OfficeHome (DomainBed-OH task); Mini-ImageNet (EasyFSL task); CIFAR-100 (USB, Solo-learn, PyCIL tasks); CIFAR-10 (Lightly, PrivacyMeter, AIF360, Opacus, PFLlib tasks); splitMNIST (Continual Learning task); IHDP (CausalML task); MNIST (ART task); SVHN (OpenOOD task); COMPAS (AIF360 task); Adult Census (Fairlearn task); TOFU / Llama-3.2-1B (Unlearning task); FML-bench repository: https://github.com/qrzou/FML-bench; 18 tasks based on publicly available research codebases across domains: DomainBed, EasyFSL, USB, Lightly, Solo-learn, Continual-Learning, PyCIL, CausalML, gCastle, ART, OpenOOD, PrivacyMeter, Opacus, AIF360, Fairlearn, Unlearning, PFLlibCode: https://github.com/qrzou/FML-benchExtracted from: pdfAgreement 57%

Explore related topics

Evaluation & Benchmarks Research Productivity Benchmarking