Ai2 Releases StateBench: Four Difficulty Levels for Tracking What a Variable Equals Right Now, and Frontier Models Drop to 13% or Lower on the Longest Inputs at the Hardest Level

What state tracking is

Reading code means knowing what a variable equals after it has been reassigned many times; reading a novel means knowing whose hand the knife passed into. This ability is called state tracking. In theory, transformers have known limits on state tracking with long dependency chains, which is one reason recurrent and hybrid architectures with a running memory, such as Gated DeltaNet, have drawn attention. But existing tests mostly focus on the hardest cases, making it hard to see exactly which level a given architecture gets stuck at.

StateBench, released by the Allen Institute for AI (Ai2) on November 3 (paper arXiv:2611.03094 by Jackson Petty, William Merrill and Ashish Sabharwal), divides the problem into four levels based on Krohn–Rhodes theory, which can be understood as kinds of variable assignment:

• r-trivial: assigned once, like let x = 4; like a birth date, you only need the first value;
• aperiodic: reassignment by value, like x = 4 and later x = 2; like a thermostat, you only need the latest setting;
• soluble: updates on the current value, like x = x + 5; like a game score, you have to carry the running total;
• general finite: assignment by reference or swapping, like x = y; the classic case is finding the ball after five cups have been swapped around (the S₅ problem).

To compare levels fairly, all four are derived from the same sequence of assignments, with the same number of variables and values and assignments at the same positions; only the type of assignment changes. The same problem is also posed in several formats, such as spreadsheet edits.

What the tests found

• Open pretrained models: 7-billion-parameter Olmo 3 (a transformer) and Olmo Hybrid (with added Gated DeltaNet layers), plus smaller models, all pretrained on the same data, nearly all did well on r-trivial, lost accuracy on aperiodic as inputs grew, and stayed at chance on general finite. Format mattered a lot: 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems but near chance on the same problems framed as spreadsheet edits.
• Frontier models: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna did well on the first two levels. On soluble and general finite, accuracy fell sharply as inputs grew longer, down to 13% or lower on the longest general finite problems. The researchers point to sensitivity as one likely reason: at the harder levels many more steps can affect the final answer, so missing a single update produces a wrong result.
• Training from scratch: the team trained two variants each of 275-million-parameter transformers, hybrids and Gated DeltaNet models directly on StateBench, on sequences of up to 256 assignments, and tested them on up to 64,000 assignments, 250 times the training length. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels out to the longest sequences; transformers learned the easier levels but degraded on longer sequences and never learned general finite. On the hardest level, the Gated DeltaNet variant with a more flexible memory update generalized the farthest, staying near perfect out to roughly 10 times its longest training sequence.

What it means

The conclusion has two parts. For Gated DeltaNet and hybrid models, the shortfall of their pretrained versions looks like a training-data problem rather than an architectural one; transformers are different, since even direct training doesn't teach them the hardest level. The researchers also list open questions, including the exact place of the restricted Gated DeltaNet variant in the hierarchy, and the fact that StateBench assumes a fixed number of variables while real conversations keep introducing new people and objects. For teams using long documents in agents or code analysis, the takeaway is that "find the latest value" and "follow a chain of changes" are tasks of different difficulty: frontier models handle the former well, while the latter is still unreliable on long inputs.