The Context Window Is a Budget, Not a Buffer

Ask most engineers what a context window is for, and you'll get some version of "it's where you put the stuff the model needs to know." That's true, but it smuggles in a bad assumption: that context is storage. Storage is passive. You can fill it up, and as long as you're under the limit, nothing bad happens — the data just sits there until something reads it.

Context windows don't work like that. Every token in the context window competes for a finite attention budget, and spending that budget on the wrong things has a real cost even when you're nowhere near the token limit. The right mental model isn't a hard drive. It's a P&L: every line item you add has to earn its place, because it's not free just because there's room.

Storage vs. budget

The distinction matters because it changes what "good" looks like. Under a storage model, the optimization is simple: fit in as much relevant information as you can, and more relevant information is strictly better. Under a budget model, more information is only better if its marginal value exceeds its marginal cost — and the cost isn't hypothetical. As context grows, precision drops and reasoning weakens across the board, not just on the specific fact that got buried.

This is the part that surprises people coming from traditional software: the cost isn't localized. Adding an irrelevant paragraph doesn't just fail to help — it degrades the model's ability to use the relevant paragraphs sitting right next to it. A model with 3,000 tokens of tightly relevant context will often outperform the same model given those same 3,000 tokens plus 15,000 tokens of marginally-related material, even though nothing in the extra material is technically wrong. The extra tokens are diluting attention that the useful tokens needed.

What "spending" actually looks like

Once you take the budget framing seriously, every context-engineering decision becomes a spend/return calculation:

• Conversation history — keeping the full transcript is the cheapest thing to implement and one of the most expensive things to carry. Every prior turn, including the ones that resolved cleanly and have no bearing on the current task, is still competing for attention on every subsequent call.
• Tool schemas — every tool definition you register is loaded into context whether or not it's used this turn. A toolset that made sense at five tools starts taxing every single inference once it grows to fifteen or twenty, which is exactly the mechanism behind the well-documented finding that agent accuracy degrades measurably once a toolset crosses roughly ten tools, and keeps sliding as it approaches twenty.
• Retrieved documents — RAG systems that return the top ten chunks "to be safe" are spending budget on the chance that one of them helps, at the guaranteed cost of diluting the ones that actually do.
• Boilerplate instructions — a system prompt that re-explains context the model already has (from tool descriptions, from prior turns, from a memory block) is spending tokens on redundant signal, which is close to the worst possible use of budget: cost with no new information.

None of these are wrong to include categorically. They're wrong to include by default, without asking what they're buying.

The P&L discipline

A useful trick borrowed from financial thinking: before adding anything to context, ask what line item it belongs to. Is it revenue (directly increases the odds of a correct answer this turn) or is it overhead (might help occasionally, mostly just sits there)? Overhead has to be justified continuously, not once. A tool that was essential to include when you built the agent six months ago might now be overhead if usage patterns shifted and it's rarely called — but it's still costing every single inference, because it never left the budget.

This is also why static system prompts age badly. A prompt written once and never revisited accumulates instructions for edge cases that mattered at the time and don't anymore, each one still taxing every call. Treating the prompt like a budget line means periodically asking which instructions are still earning their keep, the same way you'd review a subscription list rather than assuming last year's spend is still justified.

The core error is believing that "there's room" is the same question as "is it worth it." A 200K-token context window having 150K tokens free doesn't mean the next 150K tokens of context are free to add. It means you have 150K tokens of budget you can spend well or spend badly, and spending it badly costs you accuracy on the tokens you actually needed.

Where the money is best spent

If context is a budget, the highest-return spends tend to share a property: they're specific to this call, not generically useful across many calls. A retrieved paragraph that directly answers the question in front of the model is a good spend. A general company FAQ included on every call "just in case" is a bad one, even if it's occasionally relevant, because its average return across all the calls where it isn't relevant drags the whole line item down.

This is the argument for dynamic assembly over static inclusion — building context per-call based on what the current task actually needs, rather than maintaining one large context block that gets sent every time regardless of relevance. It's more engineering work upfront. It's also the difference between an agent that stays sharp as its capabilities grow and one that gets measurably worse every time you add a new tool or a new knowledge source, because you never went back to ask whether the old ones were still worth their line item.

Why "just in case" is the most expensive phrase in context design

Almost every bloated context system traces back to the same two words: just in case. A document included in a RAG result "just in case" the model needs it. A tool registered "just in case" a future task calls for it. A paragraph of background left in a system prompt "just in case" someone reads it fresh without prior context. Each individual instance feels defensible — the downside of leaving something out that turns out to matter can feel worse than the downside of including something that turns out not to. But "just in case" spending never gets revisited once the immediate anxiety that prompted it fades, which means it accumulates permanently while providing value only occasionally. A budget run entirely on "just in case" line items is, definitionally, a budget with a lot of waste in it — and the waste isn't inert, because it's actively pulling attention away from the smaller number of things that mattered on any given call.

The discipline that counters this isn't "never hedge." It's making the hedge pay rent: if something is included just in case, put an actual mechanism around it — conditional loading, a relevance check, a periodic review — rather than letting it sit in the always-on budget indefinitely because removing it once added felt riskier than adding it in the first place.

The number that should worry you

If you're building or maintaining an agent, the metric worth watching isn't "tokens remaining in the window." It's something closer to signal density — how much of what's in context on a given call is actually load-bearing for that call's decision. An agent that's technically using 40% of its context window but where half of that 40% is stale history or unused tool schemas is running a worse budget than an agent using 15% of its window where nearly all of it matters.

That's a harder number to instrument than token count, which is exactly why most teams don't track it and default to the storage mental model instead — fill it up, worry about the limit, not the cost. The teams getting real reliability gains in 2026 are the ones who stopped asking "how much can we fit" and started asking "what's this line item buying us," on every single thing that goes into the window.

---

Part of the "Context Engineering" series on aiskill.market.