AI evidence extraction

HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution

Hanhua Hong, Yizhi Li, Jiaoyan Chen, Sophia Ananiadou, Xiaoli Li, Jung-jae Kim et al. · ArXiv.org · 2026

AI-generated evidence extraction, verified across multiple analytical personas. Not a substitute for the peer-reviewed original.

9/10

Relevance

2/4

Quality (LMQS)

Evidence

Citations

0.00

FWCI

Methodology & findings

Study design

Empirical evaluation study using multi-agent framework benchmarking.

Sample

N = 110, 6 groups

Primary method

Pearson correlation coefficient (r) for meta-evaluation alignment; comparative evaluation using LLM-as-judge metrics with multiple evaluators (o3-mini-high, ChatGPT-4o-mini); percentage scoring on tree-structured rubrics and 1-5 scales; ablation study using incremental component addition.

Main result

The study demonstrates that HIRAS achieves state-of-the-art performance on experiment reproduction benchmarks, with results showing ">10% relative performance gain beyond the previous state-of-the-art using open-source backbone models and significantly reduced hallucination in evaluation." Specifically, on PaperBench-CodeDev, HIRAS achieves 64.1% accuracy with Claude-Sonnet and 57.4% with DeepSeek-v3.1, substantially outperforming prior methods. The hierarchical manager agents contribute "approximately a 10% increase" in performance through improved error correction and coordination.

Reports effect sizes.

Research paradigm

Empirical evaluation of computational system performance

Author conclusions

The authors conclude: "In this work, we present HIRAS, a hierarchical multi-agent framework for end-to-end experiment reproduction, introducing manager agents to coordinate the multi-stage workflow. Comprehensive experiments illustrate that our framework consistently outperforms prior approaches on experiment reproduction benchmarks with state-of-the-art performance achieved by open-source models, highlighting the benefits of hierarchical supervision and specialised agent collaboration across the system." They further note that "the case study underscores how hierarchical coordination improves the overall quality of reproduction and mitigates error propagation."

Risk of bias

Evaluator selection bias: different evaluators (o3-mini-high vs ChatGPT-4o-mini) used for different benchmarks; Hallucination in evaluation: original Paper2Code protocol systematically overestimates repository quality; Limited budget constraints may have prevented comprehensive evaluation across all settings; Model-specific effects: results vary substantially across different LLM backbones; Evaluator bias: LLM-based evaluators (o3-mini-high, ChatGPT-4o-mini) may have inherent biases in code assessment; Selection bias: Only 20 papers in PaperBench and 90 papers from three conferences in Paper2Code; may not represent full research diversity; Model-specific effects: Results may be influenced by choice of backbone LLMs; generalizability to other models unclear; Hallucination in evaluation: Original Paper2Code metric shown to systematically overestimate repository quality through evaluator hallucination; Evaluation cost constraints limiting evaluator coverage; Evaluator hallucination in reference-free evaluation (explicitly identified and addressed); Potential token cost bias favoring methods with less computational overhead

Limitations

The authors state: "Due to budget constraints, we do not evaluate all experimental settings with the o3-mini model
However, all reported comparisons are conducted under consistent evaluation protocols to ensure fairness across methods
In addition, our method may incur higher time and token costs than prior approaches, stemming from the increased complexity of agent reasoning and tool calling." Additionally, "the execution stage remains the primary bottleneck of current models and frameworks," with failures predominantly arising from execution errors due to "incorrect inter-file dependencies, particularly in complex directory structures."

Open questions raised

Limited work on hierarchical multi-agent systems with active supervisory roles for experiment reproduction
Need for more reliable evaluation protocols for paper-to-code generation that account for evaluator hallucination
Execution remains a primary bottleneck; inter-file dependencies and complex directory structures cause failures
Limited work addressing error propagation in fixed sequential agent pipelines
Execution stage remains a primary bottleneck requiring improved inter-file dependency management
Need for better handling of complex directory structures and cross-file coding consistency

Data: PaperBench: 20 machine learning papers from ICML 2024 with manually constructed evaluation rubrics; Paper2Code: 90 papers from ICML 2024, NeurIPS 2024, and ICLR 2024; PaperBench (20 machine learning papers from ICML 2024); Paper2Code (90 papers from ICML 2024, NeurIPS 2024, ICLR 2024)Code: https://github.com/KOU-199024/HiRASExtracted from: pdfAgreement 59%

Explore related topics

AI-Assisted Discovery Research Productivity Case Study