InferenceBench: Frontier AI Agents Flunk the Systems Engineering Test

A new arXiv paper published this week should give pause to anyone bullish on AI agents as autonomous ML systems engineers. InferenceBench 1 - a benchmark for open-ended LLM inference optimization - pits frontier coding agents against a deceptively real task: given a base model, a single NVIDIA H100, and a two-hour wall-clock budget, make that model serve faster. The results expose a meaningful gap between what agents know and what they can reliably do.

What InferenceBench Actually Tests

Most existing kernel and inference benchmarks narrow the problem to a single axis: pick the right quantization, tune a specific config, or write a faster CUDA kernel. InferenceBench takes the opposite approach.

According to the paper's official repository 2, each agent run receives a base LLM, a fixed hardware environment, and a scenario-specific objective. The agent must then autonomously deliver a running, OpenAI-compatible inference server that maximizes the target metric - while clearing both a quality gate (output correctness) and an integrity gate (no cheating by, e.g., caching golden outputs).

The benchmark defines scenarios that stress-test different bottlenecks, including time-to-first-token (TTFT), time-per-output-token (TPOT), throughput (req/s), and a balanced "all-in-one" scenario 3. Crucially, the action space is fully open-ended - agents may choose any inference framework (vLLM, SGLang, TGI, TensorRT-LLM), any attention backend, any quantization scheme, and any KV-cache layout. Wrong combinations crash on launch rather than degrading gracefully, mimicking real infrastructure brittleness 2.

The Damning Numbers

The headline finding is uncomfortable, and it is stated directly on the project's official site 3: across all scenarios, agents outperform the vanilla PyTorch baseline and most inference engines running with default configurations - so they're not helpless. But they fall short of a simple hyperparameter search over existing engine settings given the same two-hour time budget 3.

The behavioral data from the GitHub repository is even more revealing 2:

• 93.9% of agent runs ship a vLLM-based final launcher, even though SGLang, TGI, and TensorRT-LLM are explicitly named in the prompt.
• The median run launches exactly one non-default vLLM configuration over the full two-hour budget.
• 58.3% of runs pass both gates while launching at most one distinct non-default configuration, meaning continuation doesn't imply adaptation.
• Of 180 recorded runs: 65.0% pass both gates, 18.9% fail or don't complete the quality gate, 6.1% are integrity-flagged, and 10.0% fail final-server reachability or runtime checks.

The agents know the landscape - they can name frameworks, describe quantization tradeoffs, and produce valid server commands. What they struggle with is the iterative, stateful work of running, comparing, and keeping the ones that work 3.

Why This Matters for AI R&D Automation

The authors frame InferenceBench explicitly as a rung on the ladder toward AI R&D automation. In their release announcement, lead author Maksym Andriushchenko wrote that "AI R&D automation will very likely unfold gradually, starting from 'boring' tasks like inference speed optimization that are very easily verifiable (accuracy + inference time)" 4. If agents can't reliably outperform a grid search here, harder tasks - novel architecture search, full training pipeline optimization - look even more distant.

The benchmark also highlights a specific failure mode: agents default to the familiar. The near-universal collapse to vLLM, despite explicit mentions of alternatives, suggests that frontier models' strong prior toward common tools overwhelms the exploratory search behavior needed for genuine systems engineering. As the authors put it in the same announcement, agents "are not good at system-level engineering and managing complex dependencies" 4.

This connects to a broader pattern in the literature. AgentKernelArena 5 - a benchmark focused on GPU kernel optimization agents - found that existing kernel benchmarks "evaluate single LLM calls rather than full agent workflows, and none include both kernel-to-kernel optimization and unseen-configuration generalization testing" 5. AgentKernelArena was specifically designed to expose whether reported correctness and speedups transfer to input configurations the agent never observed 5. InferenceBench surfaces an analogous problem at the framework composition level: agents hardcode to what they've seen, rather than genuinely exploring the combinatorial space of inference stack choices.

Where the Field Goes from Here

InferenceBench's open-ended design is deliberately harder to game than narrow benchmarks. Unlike narrower setups "where the action space collapses to hyperparameter tuning over a known recipe," inference systems engineering forces real composition choices under brittle infrastructure 2 - and the evidence so far suggests agents are not yet up to the task.

For the community, the immediate takeaways are threefold:

1. Agents need better state tracking. The ability to remember which configurations were tried, why they failed, and what the deltas were is apparently not reliable today.
2. Diversity of exploration matters. Defaulting to vLLM 94% of the time is not a search strategy; benchmarks should penalize or at least surface this collapse.
3. Hyperparameter search is a tougher baseline than expected. Before deploying agentic pipelines for inference optimization, teams should check whether a well-structured search over existing engine settings already captures most of the gain 3.

InferenceBench is the same research group's (aisa-group) follow-up to PostTrainBench 6 - which measured CLI agents' ability to post-train base LLMs under similarly constrained real-world conditions, giving agents a single H100 and a 10-hour budget. The release announcement confirms InferenceBench is explicitly framed as "our next benchmark after PostTrainBench that measures progress on AI R&D automation" 4. The message from this new installment is clear: the agents are in the door, but they haven't yet learned how to rearrange the furniture.

---

InferenceBench is open-source. The paper is available at arxiv.org/abs/2607.20468, the benchmark code at github.com/aisa-group/InferenceBench, and the leaderboard at inferencebench.ai.