RECLAIM: New Benchmark Reveals AI Agents Can Only Reproduce 41% of ML Papers - Even With Code and Data Provided

Can AI agents do real science? Not just generate plausible-looking outputs, but actually re-run the experiments from a published machine learning paper and match the authors' reported results? A new benchmark called RECLAIM (Reproducing Claims in Machine Learning) just gave us a rigorous, sobering answer: even under the most favorable conditions, the best agents succeed less than half the time. 1

What RECLAIM Actually Tests

Reproducing a machine learning paper sounds deceptively simple. In practice, it requires installing software, parsing experimental setups from dense prose, debugging code, managing GPU budgets, and confirming that outputs match specific numerical claims - steps that span nearly the full research workflow. 2

RECLAIM formalizes this challenge as a 100-paper benchmark drawn from NeurIPS 2025 submissions. 1 For each paper, the benchmark pre-specifies which result to reproduce, what counts as success, and a hard GPU-hour budget. An agent receives the paper and whatever artifacts its authors chose to release - no more, no less. 2

The key design insight is a three-tier difficulty system driven by what authors actually published:

• Run-tier: Code, data, and model weights are released. The agent just needs to run the pipeline.
• Retrain-tier: Weights are missing, so the agent must train the model from scratch.
• Reimplement-tier: Code is missing entirely, so the agent must write its own implementation from the paper's description. 2

This tiering reflects the actual state of artifact releases in academic ML, where publication norms remain inconsistent. The benchmark was constructed from a pool of 3,414 NeurIPS 2025 papers that had an arXiv ID, narrowed down through a three-stage curation pipeline. A classifier agent read each paper, extracted its central empirical claim, and audited four artifact signals: released code, datasets, model weights, and whether datasets are standard public benchmarks. 3

The Numbers Are Stark

Four agents were evaluated, each run once per paper. The results don't flatter the state of the art: 2

The best-performing agent in each tier still fails the majority of tasks, and every agent does worst in the Reimplement tier - the situation closest to what a researcher actually faces when trying to reproduce a paper without author cooperation. 2

To put this in perspective, ML's reproducibility problem predates AI agents entirely. One sustained human replication effort reproduced only 162 of 255 attempted papers without using the authors' code, with shortfalls attributed to missing code, missing data, and underspecified experimental details. 4 RECLAIM now adds a new dimension: even automated agents with full code access can't reliably close that gap.

Why Grading Is an Agent Problem Too

One of RECLAIM's notable design decisions is separating grader from runner. A separate language model grades each run from logs and outputs, rather than trusting the executing agent's own report of success. 2 This matters because agent self-reporting is notoriously unreliable on long-horizon tasks - an agent asked to "finish the replication" can report completion before the workspace actually contains required evidence. 5

This grading architecture connects RECLAIM to broader agent evaluation design: progress and completion need to be verified by external evidence, not inferred from the agent's final message.

Living Benchmark by Design

The benchmark is explicitly built to stay current. The 100-paper set is drawn from NeurIPS 2025, but the construction pipeline is designed to be rebuilt yearly from new conference proceedings. 1 This matters enormously: agent capabilities are advancing fast enough that a static benchmark can become saturated within a year or two. By anchoring each edition to the latest NeurIPS papers, RECLAIM gives the community a repeatable, contamination-resistant track for measuring progress.

The benchmark also makes its tier composition a research variable. As the ML community improves artifact release norms - a push that NeurIPS's own reproducibility track has been driving - the mix of Run, Retrain, and Reimplement papers will shift, naturally adjusting the benchmark's difficulty profile over time.

Why It Matters

RECLAIM arrives at an inflection point. Both Anthropic and OpenAI have noted that their researchers increasingly delegate experiment-running to AI agents. 4 If agents are going to be genuine research accelerators - not just coding assistants - they need to be able to pick up a paper and reproduce its claims independently.

A 41% success rate in the easiest tier says we're not there yet. The failure cases suggest the bottlenecks are not raw coding ability but rather long-horizon coherence: managing state across many steps, correctly parsing ambiguous experimental details, and distinguishing a genuine numerical match from a superficially similar output.

For AI engineers building research-automation pipelines, RECLAIM is both a useful evaluation tool and a cautionary benchmark. Agent-assisted science is already happening; RECLAIM helps measure how much weight you can actually put on it.

The benchmark's code and data are available on GitHub, and the paper is on arXiv. 1