MirrorCode Benchmark Reshuffles Frontier Coding Models Under Harder Tests

The August update to the MirrorCode benchmark puts Fable 5 at 64% on a harder Medium and Large task set, while GPT-5.6 Sol scores 20%, GPT-5.4 scores 16%, and GPT-5.5 scores 10%. For AI builders, the important result is less about declaring an absolute winner and more about how benchmark design changes what a model is being asked to do.

The August MirrorCode leaderboard tests a different problem

The August 3 run evaluated 15 Medium and Large programs in Go and Ada, creating 30 tasks. Each task received three attempts with a 10 billion token budget per attempt. The setup excludes the easier Small programs and makes Ada a mandatory implementation language. Epoch AI warns that these results are not directly comparable with the June evaluation, which covered 25 programs, up to six languages, and a different token budget.

MirrorCode measures long-horizon autonomous coding. An agent receives a compiled binary, documentation, and visible tests, then reimplements the program without internet access, source code, or human guidance. It can query the reference binary as a black-box oracle, but its final implementation must pass both visible and hidden tests at a 100% threshold.

Ada exposes language generalization differences

The forced Go and Ada split makes language coverage a central part of the result. Ada has far less representation in common training corpora than Python, and the source reports that its corpus is roughly 235 times smaller than Python's. That makes Ada a useful stress test for models relying heavily on familiar language patterns.

Fable 5 shows the smallest reported language gap: about 64% in Go versus 61% in Ada. GPT-5.5 falls from roughly 17% in Go to 5% in Ada. The result suggests that Fable 5 is more consistent across the two languages on these long-running tasks, although it does not prove equal performance across all programming languages or software workflows.

What the scores mean for AI product teams

For teams building coding agents, the leaderboard is most relevant when work has a clear specification, executable tests, and enough scope to require sustained planning. It is a weaker proxy for ambiguous product work, architecture discussions, debugging inside unfamiliar repositories, or requirements that evolve through human collaboration.

The 10 billion token budget also matters. A large budget gives an agent more room for iterative implementation and repair, but it does not make the evaluation equivalent to a normal engineering task. Teams should compare models using their own repositories, latency limits, tool harnesses, review requirements, and cost ceilings rather than selecting from a single leaderboard rank.

Why GPT-5.5's score is not evidence of regression

GPT-5.5 scored 44% in the June paper and 10% in the August leaderboard configuration. That apparent collapse reflects a harder task mix, mandatory Ada coverage, and different evaluation rules. It should not be read as evidence that the model generally became worse.

There is also a provenance caveat. MirrorCode uses real open-source programs, so partial memorization cannot be completely ruled out, even though Epoch AI applied a memorization screen. The practical decision rule is straightforward: use the August scores as evidence about performance on this specific Medium and Large, Go and Ada, long-horizon setup. Do not treat them as universal rankings for software development.