AI evidence extraction

A case study of evaluating AI agents on a neuroscience data-to-discovery pipeline

Kai A. Horstmann, Ethan Lin, Alice A. Robie, Jennifer J. Sun, Kristin Branson · arXiv (Cornell University) · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

1/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Case study with empirical evaluation of general-purpose AI coding agents (Claude Code/Claude-Opus-4-6 and Codex/GPT-5.4) on a 7-stage fly optogenetics data-to-discovery pipeline.

Sample

< 30, 8 groups

Primary method

Pass/fail binary classification based on task-specific tolerance thresholds (ε); Continuous metric scoring: MOTA for tracking, percentile errors for keypoint estimation, balanced accuracy for classification; Comparison methods: agent score vs ground truth (Equation 1) and agent score vs scientist reference (Equation 2); Mann-Whitney U tests for statistical significance comparisons; Oracle ablation experiments with gold-standard inputs; Manual qualitative analysis of agent code iterations and failure modes; Trial-to-trial variance assessment

Main result

The study found that "agents can solve several individual pipeline stages, suggesting stage-level automation is tractable." However, "solving the end-to-end pipeline correctly requires stringing together successes across all pipeline stages, and this is beyond agents' current abilities." Specifically, "agents achieve reliable performance on several individual pipeline stages" including Feature Computation and Behavior Classification, yet "agents fail to iterate productively when self-evaluation requires scientific judgment," and they "largely fail to interpret what they see or act on it appropriately" when attempting visual inspection.

Reports effect sizes and confidence intervals.

Research paradigm

Empiricist/Pragmatist - systematic evaluation of AI agent capabilities through structured benchmarking and task design

Author conclusions

The authors conclude: "Individual pipeline stages in our benchmark are on average 10x larger (in terms of lines of code written) than those in the most closely related work, ScienceAgentBench (Chen et al., 2025) (Fig. 3), yet agents were able to solve several of them (Table 1), suggesting that stage-level tasks of the complexity studied here represent a tractable problem size for current AI agents. This is an encouraging signal for the near-term potential to alleviate bottlenecks faced by scientists through agentic automation. However, agents must solve every stage successfully to complete the full end-to-end pipeline, and the current stage-level failure mode prohibits this." They further note: "one such characteristic deserves particular attention. In science, it is common to encounter problems for which no explicit mathematical criterion has yet been formulated: the scientist has an intuitive understanding of what kinds of errors matter and why, and develops methods for visualizing the behavior of an analysis and diagnosing failure modes... Our iteration analysis exposes that current agents do attempt this behavior but largely fail to close the loop from inspection to correction."

Risk of bias

Selection bias in choice of single pipeline domain; Potential for experimenter bias in evaluation criteria design and tolerance threshold selection; Limited generalizability to other scientific domains (acknowledged by authors); Possible bias in manual code review and failure mode diagnosis; Single domain/pipeline selection bias - findings based on one neuroscience pipeline may not generalize; Limited agent diversity - only evaluated general-purpose models from two main vendors; Researcher involvement in task design - domain experts designed evaluation criteria which could introduce bias toward particular approaches; Selection of reference implementations - choice of baseline models (FlyTracker, Ctrax, JAABA) could influence evaluation outcomes; Selection bias: Single scientific domain (fly optogenetics) limits generalizability; Evaluator bias: Manual analysis of code and failure modes by researchers may introduce subjective interpretation; Task design bias: Evaluation criteria were iteratively refined based on results, potentially introducing circularity; Model selection bias: Only tested general-purpose coding agents; no domain-specific or specialized agent comparisons; Reference solution bias: Scientist reference implementations may contain bugs or design choices that agents are penalized for not replicating

Limitations

The authors state: "Our findings come from a depth-first case study of a single scientific pipeline, from which we have formed hypotheses of scientific task properties that cause the successes and failures we observe." They acknowledge that "these properties often do not correspond to a particular scientific domain, but instead to structural properties of the task or agent abilities required
The findings we describe below are directly observed in our setting
we hypothesize they will generalize best to other scientific tasks that share a high bar for precision, large-scale data, long multi-stage workflows, and reliance on data-based or visual validation
Establishing where these patterns hold across additional problems and agentic frameworks is an important direction for future work."

Open questions raised

End-to-end pipeline automation at scale remains an open challenge that scales non-trivially with task length
Agents struggle with long-context instructions and sustained attention over extended tasks
Resource management and computational efficiency for large-scale data not addressed in existing benchmarks
Generalization to large held-out and out-of-distribution data collections
Integration of vision models for visual self-evaluation and inspection-to-correction workflows
Development of stronger mechanisms for scientific judgment when explicit evaluation metrics are unavailable

Data: https://github.com/kaihorstmann/neuro-d2d-eval; Fly optogenetics pipeline data and evaluation framework available at: https://github.com/kaihorstmann/neuro-d2d-evalCode: https://github.com/kaihorstmann/neuro-d2d-evalExtracted from: pdfAgreement 62%

Explore related topics

Evaluation & Benchmarks Research Productivity Case Study