Leading AI Models Score 13% or Lower on Test of Tracking Changes in Long Texts

StateBench, a benchmark that measures how well AI can follow information as it changes over the course of a long text, was released on November 3. The researchers tested Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna. On the hardest task type, all four models dropped to 13% accuracy or lower once inputs reached their maximum length, the researchers reported.

Is finding the last update enough?

StateBench tests whether a model can work out the current state of something described in a text. It is the skill needed when an object changes hands in a novel, or when a value in code is modified several times over. Some questions can be solved simply by locating the latest update; others only yield an answer if every preceding change has been tracked without a single omission.

The benchmark is divided into four levels according to how complex the changes are: finding an initially assigned value, finding the most recently changed value, repeatedly adding changes on top of an existing value, and tracking the current value of another item as well. The hardest level resembles the shell game of shuffling five cups and then identifying which one hides the ball. To make the levels comparable, the researchers held the number of items and values, as well as the positions of the modifications, constant, varying only the manner of modification.

Where models break down as texts grow longer

All four models performed well on questions asking for a value that is set once or for the most recently assigned value. On questions requiring them to accumulate changes or to follow relationships between items, however, accuracy fell sharply as inputs grew longer. The researchers pointed to one likely factor: the harder the level, the more intermediate steps feed into the final answer, so missing even a single change along the way easily produces a wrong result.

Source: Google (press release)

Open models tested separately were also thrown off by how the problems were presented. Some smaller models scored near perfectly on one format of the same fixed-value question. Yet when the identical problem was presented as a spreadsheet edit history, they performed no better than random guessing. The researchers said they built the same problems in multiple formats precisely to expose such gaps.

Can training solve it?

To separate the effect of training from the effect of architecture, the researchers trained six models from scratch on StateBench problems. During training, models saw up to 256 value-setting operations; at test time that was stretched to as many as 64,000, or 250 times the training length. The comparison included the widely used transformer architecture, the Gated DeltaNet architecture, which continuously updates its memory, and a hybrid of the two.

Source: Google (press release)

In this experiment, the Gated DeltaNet and hybrid models held near-perfect scores on the first three levels all the way up to the longest inputs. On the hardest level, the Gated DeltaNet model, with its more flexible memory updates, answered almost perfectly up to roughly 10 times its training length. The transformer, by contrast, degraded on even the easier levels as inputs lengthened, and could not solve the hardest level even when trained on it directly, the researchers reported.

The results suggest that the ability to locate information in a text and the ability to follow a series of changes through to the end are two different things. That said, StateBench only covers situations where the number of items to be tracked is fixed from the outset. Cases in which a new person or object enters partway through a long conversation fall outside the current scope of the evaluation, the researchers explained. Everyday life and work look more like those conditions. Whether AI can keep tracking changes to the end in such settings is the question that remains to be answered.