All posts
IndustryOctober 16, 20263 min read

The Clock Doesn't Stop: RealtimeBench Exposes a Fatal Flaw in How We Test AI Agents

A new benchmark from Peking University and Samsung Research reveals that the most capable LLM agents can completely collapse under a live clock - and proposes a two-timescale self-improvement method to fix it.

Every standard AI benchmark shares a quiet, convenient assumption: the world holds its breath while the model thinks. Submit a question, wait for the answer, score the result. It's clean, reproducible - and, according to a paper published this week, potentially misleading about whether our agents can operate in the real world at all.

The paper, The World Won't Wait: Evaluating and Teaching Agents Under a Live Clock (arXiv:2610.20665), introduces RealtimeBench and a companion training algorithm called TORSI, together making a pointed argument: a correct action that arrives too late is, in practice, an incorrect action. 1

What RealtimeBench Actually Measures

Authored by researchers from Peking University and Samsung Research Beijing, the work introduces a benchmark of 180 tasks spread across six classic game types - including Mario, Snake, and Tetris. 2 The twist is deceptively simple: while the model is generating its response, the game keeps running. The last-issued control stays active, and time ticks on. 2

The same suite is also run in a paused mode (the standard assumption) and an accelerated mode, giving three matched conditions that cleanly isolate the effect of the live clock. 2

The results are striking. GPT-6.1-sol, a highly capable but slow reasoner averaging roughly 19 seconds per response, outperforms the faster Jev model across five of six game types when games are paused. When games keep running, their average scores are 1.51 and 13.51, respectively - a near-total collapse for the slow reasoner. 3 Yet even the faster model is far from competent at real-time tasks: no evaluated model averaged more than 16.62 out of 100 on the live-clock condition. 3

This reveals a fundamental tension the authors call the strong reasoning vs. fast response problem: the cognitive depth that makes frontier models impressive on static benchmarks actively works against them in time-sensitive environments, because every token of reasoning is time the world uses to move on. 4

TORSI: Teaching Across Two Timescales

The paper doesn't just diagnose the problem - it proposes a solution. Two-Timescale Offline Recursive Self-Improvement (TORSI) separates deliberation from execution. 5

The architecture works like this:

  • A slow teacher studies failures and tests corrections in paused game sessions between live attempts, then distills what it learns into reusable written instructions.
  • A fast student acts during live gameplay using those instructions - crucially, without calling the teacher at all during a run.
  • Validation scores gate which updates get adopted, preventing regressions from being written into the instruction set. 5

After two TORSI teaching rounds, Jev's average test score climbs from 13.51 to 24.19 on the 0 - 100 progress scale, with improvements across all six game types. 6 Even more encouraging: the generated instructions transfer - they improve average scores for three other models without any retuning, though some game-specific regressions persist. 6

A Broader Crisis in AI Evaluation

This paper arrives amid growing consensus that the benchmark ecosystem itself is under stress. A companion paper published the same day, Evaluating Model Capabilities After Benchmark Saturation (arXiv:2610.20667), documents how rapid capability gains are rendering standard benchmarks unable to distinguish between frontier models. 7

Once multiple models approach perfect scores, a benchmark loses its power to inform AI safety cases or frontier deployment decisions. 7 The paper finds that pairwise judging of model transcripts reduces prediction error by 5.3× compared to relying on saturated benchmark scores alone - and that capability-relevant information is concentrated specifically in reasoning traces and tool-call logs, not just final answers. 7

This connects directly to what RealtimeBench is attempting: standard static accuracy metrics simply don't capture whether an agent can operate in time-pressured, dynamic conditions. The two papers together suggest the field needs to rethink evaluation along at least two axes - the clock, and what we read from transcripts.

Why It Matters

Most deployed AI agents don't interact with patient, frozen environments. A customer-service bot handles parallel requests. An autonomous driving assistant faces a world that never waits. A trading agent operates on millisecond windows. The assumption that we can evaluate these systems on paused environments and then deploy them into live ones is quietly dangerous.

RealtimeBench provides a concrete, reproducible framework for measuring the gap between static and real-time capability - and TORSI offers a first principled approach to closing it without requiring online teacher calls during deployment. Both tools are timely contributions as the community grapples with what "capable" really means when the clock is ticking.

The benchmark and paper are available at arxiv.org/abs/2610.20665.

This article was researched and drafted by an AI writer agent (claude-sonnet-4-6) and reviewed by an editor agent before publishing.

Ask about Julian Walder

Grounded in his real work

Hi! I'm Julian Walder's assistant. Ask me anything about his work, projects, or background in AI.