Context windows are not a data strategy

Every few months a model vendor announces a bigger context window, and every few months someone on an AI steering committee asks whether that means the data problem is solved. It doesn't, and the arithmetic is worth doing out loud.

The arithmetic

Take a mid-size enterprise: a few thousand employees, a few million customers, a few hundred operational systems. The live state an agent might plausibly need to reason about (current orders, inventory, account status, open tickets, recent events, policies) runs to hundreds of gigabytes once you stop counting history and only count now.

The largest generally available context window holds, generously, a few hundred megabytes of text. That is about 0.3% of the live state. Even if windows grow tenfold, you are at 3%. The context window is a working memory, not a data layer, and treating it as one is how teams end up with agents that are eloquent about things they haven't seen.

Three ways teams fill the gap, and what each costs

1. Stuff the prompt. Dump everything that might be relevant into context on every call. Works for demos. In production the token bill scales with the size of your business, latency climbs with it, and the model's attention degrades on the long tail. Several teams we talked to retired this approach within a quarter.
2. Retrieve from a copy. Vector search over an index refreshed on a schedule. Excellent for documents and policy; dangerous for operational state, because the index is always some minutes or hours behind reality. The agent answers from a snapshot of a world that has moved.
3. Read from the live stream. Give the agent tools that query the current state through a governed access layer, with freshness measured in milliseconds. More engineering up front, and the only approach where the agent's view of the world and the world agree.

The decision that actually matters

Model choice gets the board slide. Context length gets the press release. But the decision that determines whether an agent program works is quieter: where does the agent get the 99.7%?

"We spent six months evaluating models and six days deciding how agents would read our data. We had it backwards."

Teams that get this right build a data access layer that is model-agnostic. Swap the model next quarter, keep the plumbing. Teams that get it wrong rebuild their data integration every time the vendor landscape shifts, which in 2026 is roughly monthly.

What "enough context" actually means

Enough context is not the most context. It is the right context, right now, with a record of what was served. An agent that reads 40 KB of fresh, relevant, governed state will outperform one handed 2 MB of stale everything, and it will cost a fraction to run.

Bigger windows are good. They are not a strategy. The strategy is the layer that decides what goes in them.