RealMath-Eval: Why State-of-the-Art LLM Judges Fail on Real Human Math Reasoning

The AI community has developed a convenient shorthand for evaluating language models: use another, more powerful LLM as the judge. It's fast, it scales, and on many synthetic benchmarks it correlates respectably with human assessments. A paper posted to arXiv on June 10, 2026 - RealMath-Eval (arXiv:2606.10254) 1 - throws cold water on that optimism, at least where mathematics is concerned. Its central claim is blunt: SOTA judges struggle with real human reasoning in ways that sanitized benchmark problems have been hiding.

The Core Problem with Current Math Evals

To understand why this matters, you need to appreciate how lopsided today's math benchmarks are. The vast majority rely on competition-style problems (think AIME, AMC, and MATH dataset entries), formal proof corpora, or carefully curated challenge questions designed to be hard in a particular, structured way. As the closely related RealMath benchmark (arXiv:2505.12575) observes, existing benchmarks "rely primarily on competition problems, formal proofs, or artificially challenging questions - failing to capture the nature of mathematics encountered in actual research environments." 2

Real mathematical work looks nothing like that. Working mathematicians, students, and engineers produce reasoning that is exploratory, sometimes nonlinear, full of intermediate dead-ends, and expressed in heterogeneous notation. When a practitioner solves a problem, they may sketch three approaches, abandon two, and arrive at the right answer via a route no benchmark designer anticipated. As RealMath notes, mathematics encountered "in the wild" - particularly in research settings - "differs substantially from competition problems in structure and topics, rarely relies on formal proofs, and considers statements and results that are not (exclusively) designed to be maximally challenging." 2

RealMath-Eval is designed to surface exactly this gap. Rather than using synthetic or competition-curated problems, the benchmark draws from authentic human mathematical activity - the kind documented in research papers and mathematical forums where real problem-solving in the wild is recorded. 1

What Makes a Judge Fail Here?

The failure modes of LLM-as-a-Judge on math are increasingly well-documented in related work, and RealMath-Eval probes them. The following failure patterns are consistent with the paper's framing and the broader literature on LLM evaluation:

Surface-level pattern matching. LLM judges are trained to recognize reasoning that looks like strong math reasoning. If a student produces a formally unusual but correct derivation - one that doesn't match the stylistic fingerprints of high-scoring competition solutions - judges are biased toward marking it down, effectively evaluating style as much as substance. Research on how reasoning chains influence LLM judgment has found that "weak judges are easily swayed by reasoning presence, frequently accepting incorrect answers accompanied by fluent reasoning." 3

Sensitivity to intermediate errors. Real human reasoning often contains self-corrections and exploratory missteps before landing on the right answer. Automated judges trained on clean chain-of-thought data are poorly calibrated for this. They may penalize the presence of a wrong sub-step that is subsequently corrected - a behavior that a human grader would recognize and reward.

Verification asymmetry. One of the hardest unsolved problems in LLM evaluation is knowing whether a judge is actually checking a claimed derivation or merely pattern-matching against likely-correct-looking prose. For mathematical reasoning specifically, this distinction is critical. The RealMath benchmark explicitly identifies "enabling reliable automated evaluation through verifiable statements" as one of the central design challenges in authentic math evaluation, and directly notes that "using LLMs as judges introduces reliability concerns." 2

Contamination masking real difficulty. Competition-math benchmark scores for frontier models have been creeping toward saturation. A common trend in recent benchmarks has been to design problems that are "as challenging as possible, resulting in near-zero performance" 2 - yet on older, widely-used benchmarks, top models now post very high scores. This creates the illusion that math reasoning is largely solved. RealMath-Eval pushes back: difficulty in authentic math settings doesn't come from synthetic hard problems, but from the open-ended, contextual nature of real work. As RealMath's authors note, performance increases on research-level tasks "may be more indicative of the utility of LLMs on common mathematical research tasks" than competition leaderboard positions. 2

The LLM-as-a-Judge Problem Is Bigger Than Math

The implications of RealMath-Eval reach beyond mathematics. The "LLM-as-a-Judge" paradigm is now used pervasively - for RLHF reward modeling, benchmark leaderboards, post-training quality checks, and production system evaluation. If the judges are systematically miscalibrated on authentic human reasoning patterns in a verifiable domain like mathematics, the calibration failures in softer domains (legal reasoning, scientific analysis, creative work) are likely even larger and harder to detect.

Related research on legal benchmarks makes this pattern visible in another domain. The LEXam benchmark (arXiv:2505.12864, accepted at ICLR 2026) 4, which tests models on 340 real law school exams covering 116 courses, finds that current LLMs "notably struggle with open questions that require structured, multi-step legal reasoning." 4 Crucially, LEXam's authors found that making LLM judge evaluation reliable required deploying "an ensemble LLM-as-a-Judge paradigm with rigorous human expert validation" - not simple prompting - to "demonstrate how model-generated reasoning steps can be evaluated consistently and accurately, closely aligning with human expert assessments." 4 The same pattern - judges that work on structured tasks but degrade on authentic human problem-solving - appears to be domain-general.

A parallel thread of research on frontier reasoning models reinforces the concern from a different angle. Work on procedurally generated reasoning stress tests (arXiv:2507.07313) 5 finds that although state-of-the-art LLMs achieve remarkable performance on challenging competitive math and coding benchmarks, "they also frequently fail on tasks that are easy for humans." 5 The authors identify a key failure mode they term "reasoning delirium," in which LLMs "tend to 'overthink' easy problems, often erroneously reusing reasoning steps corresponding to the more complex puzzle solutions." 5 If models themselves misjudge task difficulty this way, LLM judges evaluating those same models face a compounded calibration challenge.

What Should Engineers Take Away?

A few concrete lessons for teams building on LLM-as-a-Judge pipelines:

1. Benchmark your judge on authentic data. If your judge was validated only on synthetic or competition-style math problems, you don't yet know how it performs on the actual distribution of work your users produce.

2. Treat judge scores as uncertain on open-ended tasks. Competition-math accuracy and judge agreement on natural mathematical writing are different things. Don't conflate leaderboard performance with real-world math evaluation quality.

3. Invest in verifiable evaluation design. The most robust mitigation is to structure problems so correct answers can be deterministically verified - reducing reliance on the judge for final correctness assessment and using it only for partial-credit or reasoning-quality dimensions. This is the approach taken by both RealMath 2 and LEXam 4.

4. Watch saturation signals carefully. When benchmark scores approach ceilings, it's often a signal that the benchmark no longer reflects authentic difficulty - not that the model has mastered the domain.

Why It Matters

The proliferation of automated evaluation pipelines has made it easy to forget that the evaluator itself is a model with failure modes. RealMath-Eval joins a growing body of work - from the RealMath benchmark for research-level math 2 to procedurally generated reasoning stress tests showing frontier LLMs frequently fail on tasks that are easy for humans 5 - that is systematically mapping where automated judges go wrong. As model capabilities advance and the tasks we deploy them on grow more complex, the quality of evaluation infrastructure will increasingly determine whether we're measuring genuine improvement or just getting better at gaming our own metrics.

Getting evaluation right isn't a footnote to the AI capability story. It's load-bearing infrastructure.