peekgenticBook a call

News analysis · AI reliability

AI models lose track in long documents, a new test finds. What it means for your office

Ai2's new StateBench test found top AI models handle facts and latest values well, but fall apart on long chains of changes, down to 13% or lower on the longest hardest problems. Your running totals belong in your ERP, not in a chatbot's head.

By Miguel GutierrezCo-founder, Peekgentic · Dallas
· 4 min read
Preview card: AI models lose track in long documents.
Finding a fact is one job. Following a long chain of changes is another.Peekgentic

In brief

A test published Nov. 3, 2026 by the Allen Institute for AI found that four top AI models do well at finding a fact or the latest value in a long document. But they break down on long chains of changes, scoring 13% or lower on the longest, hardest problems. It's a lab test, not a test of office paperwork, and it isn't a reason to drop AI. It is a good reason to keep running totals and order status in your ERP, and to have AI read one change at a time.

  • Ai2's new test found four top AI models did well at finding facts and latest values in long inputs.
  • On long chains of changes, accuracy fell sharply, to 13% or lower on the longest, hardest problems.
  • It's a lab test with generated puzzles, not office paperwork, and a single PO is far shorter than the lengths where scores collapsed.
  • The fix is design: keep running totals in your ERP, have AI read one change at a time, and test on your own long cases.

What happened

The Allen Institute for AI (Ai2), a research nonprofit, published StateBench on Nov. 3, with a technical report by researchers from Ai2, New York University and the Toyota Technological Institute at Chicago. It measures something called state tracking: can a model keep up with how things change as it reads a long document?

The test has four levels, and each one is harder to follow. Ai2 ran them on four current top models: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna.

All four did well on the two easier levels. On the two harder levels, accuracy fell sharply as inputs got longer. On the longest problems at the hardest level, it dropped to 13% or lower. The report says the models fell to chance-level accuracy once inputs reached between 10,000 and 64,000 tokens. On short inputs, the report's chart shows them doing far better. None of the four escaped the drop, though they didn't all fall at the same point.

A token is a chunk of text. Anthropic's glossary puts one at about 3.5 English characters on its earlier models, and says newer ones use more tokens for the same text. By our rough math, 10,000 tokens is a few thousand words: a long email thread or a stack of documents, not a single purchase order.

Why an owner should care

The levels sound academic, but they look a lot like office work. The everyday examples in the middle column are Ai2's. The office examples on the right are our own reading. Ai2 didn't test any of them.

The four StateBench levels, in office terms

LevelAi2's exampleIn your office (our reading)Top models, long inputs
A value that never changesA birth dateA customer's account numberDid well
The latest value winsA thermostat settingThe newest promise date on an orderDid well
Each change builds on the lastA game scoreOpen quantity after partial shipments, or a running balanceFell sharply
One value depends on anotherA ball under swapped cupsParts moved between vans and jobs, or "same date as line 7"Fell sharply, to 13% or lower at the longest

The pattern matters more than the score. If you ask a model to find something in a long thread, you're mostly in the safe rows. If you ask it to work out where things stand after a long history of changes, you may be in the risky ones. And its answer will look just as confident either way.

A second paper points the same way. LEAPS, posted Oct. 19 by a different group, tested 30 models. It found that models near perfect at recalling facts can vary widely at state tracking, and degrade much more as inputs grow. It also found that "thinking" models often did worse than their base versions, sometimes by running out of budget or oversimplifying the task.

Ask AI to read the change. Let your system of record do the math.

Where the other side has a point

This test shouldn't scare anyone off, and there's plenty it doesn't show.

  • It's a lab puzzle. StateBench uses generated sequences of changes, shown in formats such as code and spreadsheet cells. It doesn't use real POs or invoices. Real paperwork has context that may help or hurt.
  • It tests reading, not a whole system. The report describes models reading a sequence and writing out the final state. We didn't see any mention of models getting a calculator or a database. A well-built system wouldn't ask a model to do this from memory.
  • Most office documents are short. One PO or invoice sits far below the lengths where these scores collapsed. Reading one document is a different job.
  • Models change quickly. Ai2 found that newer recurrent and hybrid designs, when trained directly on these tasks, did almost perfectly on the first three levels at the longest lengths tested. That's research, not a product you can buy, but this gap may narrow.
  • Ai2 itself sees gaps in the test. It uses a fixed set of variables, while real work keeps adding new ones. Format also moved scores on the open models it tested, so your paperwork may behave differently from either result.

What changes in your office, and what doesn't

What doesn't change: using AI to read an email or PDF, pull out the fields and draft an entry still makes sense. Nothing here argues against it, and the same goes for finding a fact in a long contract.

What does change is where you let the running answer live. Here's what we'd do:

  1. Keep state outside the model. Open quantities, balances, stock and order status belong in your ERP or a database. The model reads one change, and ordinary software applies it to the current record. That's how we'd handle revised POs and change orders.
  2. Don't paste the whole history. Skip the six months of email and the "where does this order stand?" question. Give the model the current record from your system, plus the one new message.
  3. Test it on your own long cases. Find a closed job with a messy thread, where you already know the final answer. Ask your tool for the final quantities and dates. Then compare. A short demo won't show this problem, so use the long, ugly ones.
  4. Ask vendors the direct question. "Where is the running total stored, and what does the AI do if it doesn't match?" A good answer names a system of record and a check, and doesn't just point to a bigger memory.
  5. Keep a person on what matters. Before a changed order, a payment or a customer promise goes out, someone should see the change next to the current record. See our guide to keeping a person in the loop.

This also applies to staff using a chatbot on their own. If someone on your team pastes a long spreadsheet history into ChatGPT or Claude and asks for a final count, they should check it against the source. A spreadsheet formula is still the better tool for adding things up.

Common questions

What is StateBench?

A benchmark from the Allen Institute for AI, published Nov. 3, 2026, that tests how well language models keep track of changing information across long inputs. It has four levels of difficulty, from values that never change to values that depend on other values.

What did it find about ChatGPT, Claude and Gemini?

Ai2 tested GPT-6 Sol and Luna, Claude Opus 5.5 and Gemini 3.8 Flash. All did well on the two easier levels. On the two harder levels, accuracy fell sharply with length, reaching 13% or lower on the longest problems at the hardest level.

Does this mean AI can't handle my purchase orders?

No. The test uses generated puzzles, not real office documents, and one PO is far shorter than the lengths where scores collapsed. The caution is about asking a model to work out a running result across a very long history.

What is state tracking in plain English?

Keeping up with how something changes. It's knowing an order's open quantity after five partial shipments, or which van a part is on after it has been moved three times.

How do I protect against this?

Keep running numbers in your ERP or a database. Have AI read one change at a time, and let ordinary software apply it. Test your tool on long, real cases where you already know the answer, and have a person approve changes that matter.

Will newer AI models fix this?

Possibly, in part. Ai2 found that recurrent and hybrid designs trained directly on the task did much better, and calls that a direction for future research. Even if models improve, your ERP remains the record you can audit.

Sources

  1. Ai2, StateBench: Testing how language models track state changes over long documents, Nov. 3, 2026
  2. Petty, Merrill and Sabharwal, A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models, arXiv, Nov. 2, 2026
  3. Yen et al., LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking, arXiv, Oct. 19, 2026
  4. Anthropic, Glossary (tokens), checked Nov. 5, 2026

More from Peekgentic

Published November 5, 2026 · Written by Miguel Gutierrez, Co-Founder, Peekgentic (Dallas, TX). Every statistic links its source.