It Scores the Model *and* the Harness — Which Is What You Actually Buy
Read the leaderboard line carefully: 38.8% belongs to "Fable 5.1 running on Claude Code," and 33.8% to "Astra running on Codex CLI." Real-SWE scores the model together with its execution framework rather than rating a bare model. That framing is closer to reality. Nobody calls a naked model in production — you use an agent tool, and it decides how the codebase gets read, how work is broken into steps, how retries happen, and what gets cut when the context fills. The same model on a different harness can produce a completely different result. Most public leaderboards abstract that layer away; Real-SWE puts it back. One more design choice matters just as much: the tasks come from licensed private production codebases — examples include an app with over 200,000 users, a fintech platform processing more than 100,000 bank statements, and enterprise sales tooling. Specific Labs estimates roughly 99% of tokens inside real enterprises are invisible to frontier models. Whatever fluency a model has on open-source repositories does not reach the part of the code that lives in your company.
The Aggregate Is Nearly Useless; the Per-Task Variance Is the Story
The finding to leave with is how wide the spread is. Grok 4.6 went eight-for-eight on a customer identity migration where second-place GPT-6 Astra managed 1/8. On the S3 datastore measurement task, the top three pairs went 0/8 collectively, while GLM 5.3 — not in the top three overall — scored 3/8. Which means picking a tool by "who is first on the aggregate" is close to meaningless. The workable alternative is direct: take three to five tasks most typical of your team, run each candidate pair eight times on them, and read the pass rate and failure shape rather than someone else's ranking. That is exactly how Real-SWE is built — 80 rollouts per pair, not one run and a verdict. There is also a counterintuitive number: rollouts under ten minutes failed 71.4% of the time, longer ones 73.4%. Running longer did not make success more likely, so "let it think a while longer" is not the cure for this failure class.
The Biggest Failure Isn't Wrong Code — It's Missed Requirements
Of 468 failed rollouts, 190 (40.6%) were classified as missed requirements, the single largest bucket and more common than logic errors. This is the most directly actionable finding. If your agent frequently "finishes but it's wrong," the problem is more likely requirement transmission than model capability. Practical moves: write acceptance criteria as a checklist that can be verified item by item, have the agent restate its understanding before it starts, and state explicitly which behaviors must remain unchanged after the edit. A more expensive model does not fix unread requirements. This site covered IFM publishing the reward-hacking contamination in its own Terminal-Bench score on September 7. That approach audits the score; Real-SWE instead swaps in code the models have not seen. Both point at the same gap: between a coding score on a public leaderboard and "can do work inside your company's codebase" lies a distance nobody has been measuring. The limits deserve repeating in the team's own terms: this does not prove public benchmarks are inflated by contamination, and ten tasks is a small sample. Even so, the leader failing more than 60% of the time on private code is a fact that does not depend on sample size.
via: The New Stack, Specific Labs: Real-SWE benchmark page, Hacker News discussion