Speculative Macro Commit: Cutting LLM Agent Latency by Nearly Half With Multi-Step Action Prefetching

The latency problem lurking inside every production tool-using LLM agent is deceptively simple: the model can only decide what to do next after it sees what the last tool returned. That serialization forces a stop-and-wait cycle - infer, call tool, wait, infer again - and the cumulative drag can make even modest agentic tasks feel glacially slow. A paper accepted at MLSP 2026 and posted to arXiv on September 4 introduces Speculative Macro Commit (SMC), a runtime mechanism that borrows a trick from computer architecture to break that cycle at the multi-action level. 1

The Core Problem: Serial Action - Observation Turns

Tool-using LLM agents spend wall-clock time not only on model inference but also in serial action - observation turns, where each tool call, environment transition, and observation can delay subsequent decisions. 1 Every round trip adds latency that compounds across the dozens of steps a complex task might require. Prior work on "speculative actions" addressed this at the single-step level - predicting one future tool call and executing it speculatively while the main model is still thinking - but multi-step chains remained sequential. SMC targets that remaining gap.

How SMC Works: Two Tiers and a Macro Library

SMC introduces a two-tier agent architecture. 1 A large authoritative actor model produces the official trajectory, while a faster speculative drafter model continuously predicts and executes future action chains on an isolated environment snapshot. [1, 2] Crucially, the drafter doesn't operate blind.

SMC mines recurring multi-action skeletons from training traces and stores them in a macro library, which is used to match against action chains predicted by the drafter at runtime. 2 Think of it as a pattern cache: when the agent encounters a recognizable sequence opener (say, "look up a customer, then fetch their orders, then check inventory"), SMC can pre-execute the whole chain in the background before the authoritative model has even decided to start it.

The commit logic is elegant: when the actor's next tool call matches the first drafted action, SMC commits the remaining pre-executed draft steps, together with their observations, to the official trajectory. 3 If the draft diverges, it's discarded and the actor proceeds normally - no correctness is sacrificed.

Benchmark Results

The researchers tested SMC using Qwen3.5-27B INT4 as the authoritative actor and Qwen3.5-4B as the speculative drafter. On the τ²-Bench Telecom subset, SMC matched the sequential agent's overall accuracy while reducing latency by 10.23% over the prior Speculative Actions (SA) baseline and 18.59% over sequential execution. 4

The more dramatic number comes from AppWorld, a mobile-app sandbox benchmark requiring complex multi-step API interactions: SMC reduces wall time by 7.7% over the SA baseline and 44.9% over sequential execution, with only a small reduction in task completion rate. 5 Nearly halving wall-clock time on a real agentic benchmark is a meaningful result - the kind that shows up in production SLAs, not just leaderboards.

Why Multi-Step Speculation Beats Single-Step

The key insight is that single-step speculative actions leave significant latency on the table whenever tasks involve recurring multi-action patterns. By mining training traces for common action skeletons and reusing them as macros, SMC effectively amortizes speculative execution across entire sub-task sequences rather than individual tool calls. 6 The macro library acts as learned structure - the drafter isn't just guessing the next action, it's pattern-matching against known task shapes.

This is conceptually similar to how branch predictors in CPUs evolved from single-branch prediction to trace caches that speculatively execute entire basic-block sequences. The parallelism dividend compounds with chain length.

Connection to the Broader Agent Latency Landscape

SMC joins a growing family of speculative execution approaches for agentic systems, including the original Speculative Actions framework it explicitly benchmarks against. The code is publicly available on GitHub 7, which positions SMC as a practical drop-in layer rather than a research artifact. The two-tier architecture - large authoritative model plus small fast drafter - is also hardware-friendly: the small model can run on CPU or a separate GPU while the large model handles the critical path.

Why It Matters

LLM agent latency is quickly becoming a first-class engineering problem. As agents take on longer-horizon tasks - browsing, coding, customer service workflows - the per-step overhead multiplies. Techniques like SMC are important because they attack this latency structurally, without requiring model retraining, prompt engineering changes, or more expensive hardware. A nearly 45% wall-clock reduction on AppWorld, at parity accuracy on τ²-Bench, suggests that speculative multi-step execution is ready to graduate from research novelty to production consideration. For any team running tool-calling agents in latency-sensitive settings, SMC is worth a close read.