Splitting 100M Tokens Across Parallel Sessions Beats One Long Run by 264 Elo: a Paper Measures the Point Where Agents Stall

The paper "When Agents Slow Down" (arXiv:2609.15309), submitted September 14, proposes Elo-per-token analysis: track the best solution found at each token budget, then use a Bradley-Terry model to aggregate within-task orderings into Elo ratings comparable across tasks — addressing the problem that agents allocate test-time compute adaptively, which makes performance scaling hard to measure. It runs four general-purpose agents across four open-ended benchmarks with single sessions up to 100M tokens, plus three feedback-driven optimization harnesses under controlled intervention. The core result: against independent sampling as a theoretically characterized reference (400 Elo per tenfold increase in compute, growing linearly with log compute), agents initially convert tokens into Elo faster than the reference, but their marginal gains diminish and eventually fall below it — while on shared AtCoder Heuristic Contest tasks, the strongest historical human contestants improve superlinearly over contest time. The authors define the scaling inflection point as the per-session budget where marginal gains match the reference, and on FrontierCS Polyomino Packing splitting the same 100M tokens across parallel sessions gained 264 Elo over one long session and 355 over ten short ones. Authors include Kaiyuan Liu, Qiuyang Mang, Luke Zettlemoyer, Alex Dimakis and Alvin Cheung; the preprint has not been peer reviewed.

It Delivers More Than a Conclusion — It Delivers a Knob You Can Measure

"Agents stall when you run them long" is not a new observation, but this paper turns it into a quantity. The methodological key is choosing a reference with a theoretical characterization: independent sampling — draw repeatedly and independently, keep the best — whose Elo grows linearly with the logarithm of compute, at 400 Elo per tenfold increase. With that baseline in place, "does the agent's orchestration actually add value" becomes a decidable question. The answer is piecewise: early on, yes, the agent converts tokens into Elo more efficiently than independent sampling; later, no — marginal gains decline steadily and eventually drop below the reference. Past a certain budget, continuing to run the same session is worse than simply drawing more independent samples. That crossing point is what the paper defines as the scaling inflection point: the per-session budget at which marginal Elo gains match the independent-sampling reference.

Neither Longer Nor More Fragmented Is Better

The most practical figures come from the splitting experiment on FrontierCS Polyomino Packing. With the same 100M-token total budget, spreading it across parallel sessions gained **264 Elo over a single long session** and **355 Elo over ten short ones**. Both extremes are ruled out. One continuous run pushes past the inflection point and wastes the back half of the compute. Slicing too finely ends each session before it reaches the stretch where the agent is most efficient. There is a measurable optimum in between, and finding it starts with locating the inflection point on your own tasks. For anyone paying per token to run agents, that translates directly into practice: **don't start by asking whether to give it more time — measure where the inflection point sits for your class of task, then set per-session budget and parallelism from that.** It is cheaper than either extending the thinking or running many attempts and taking the best, and it is a parameter you can measure once and reuse.

The Human Baseline Is the Weightiest Part

The paper does not stop at comparing agents to each other. On shared AtCoder Heuristic Contest tasks, the strongest historical human contestants improve **superlinearly** over contest time — getting better faster as they go, while agents get better more slowly. The authors' blog post makes it concrete: in a two-week coding marathon, current agents flatten out around the 24-hour mark even when a single trial consumes up to 100M tokens, while top humans start slower and eventually pass them by a wide margin. This is not a sentimental "humans are better." It quantifies the gap in continual learning: humans change their method mid-process, while an agent within one session is largely repeating and fine-tuning. The authors read this optimistically — there is substantial headroom after agents slow down — but it is headroom that has to be won with new test-time strategies rather than a bigger budget. This site covered OpenAI's internal figures on September 8: 3.1 agent workdays per human workday by mid-August, with more than half of successful four-to-eight-hour tasks requiring at least one human intervention. This paper supplies a mechanism for that observation — the issue is not the timing of intervention but that long sessions stall on their own, which is why a person has to step in at some point. The limits, in the authors' own terms: a preprint, not peer reviewed; four agents across four open-ended benchmarks, with the human comparison drawn from historical AtCoder Heuristic Contest contestants. The inflection point's location varies by task and agent — what the paper offers is a method, not a universal number.

via: arXiv:2609.15309, "When Agents Slow Down", Hugging Face paper page, author blog, "Humans Still Beat AI in the Long Horizon"