A Context Engineering Checklist for Your Next Agent

Every idea in this series points at the same underlying claim: most agent failures trace back to what the model was shown, not what the model is capable of. That's a useful thing to believe in the abstract. It's more useful as something you can actually check against a specific agent before it ships, rather than after it's already misbehaving in production and someone's filing a ticket to swap the model.

What follows is the checklist version — nine questions, each one tied to a specific failure mode that shows up repeatedly once agents move from demo to production. None of them are exotic. All of them are questions teams skip specifically because the agent works fine in the narrow conditions of a demo, and every one of these failure modes is a scaling problem that a demo is too small to expose.

A demo runs a handful of scripted paths, with a short conversation history, a small toolset, and a knowledge base that hasn't yet accumulated the noisy, near-duplicate content real usage produces over months. Every one of the nine failure modes below needs volume or duration to show up — dozens of tools, hundreds of turns, a memory store with real accumulated history — which is exactly what a pre-launch demo doesn't have. That's not a reason to skip the checklist before shipping. It's the reason the checklist matters most before shipping, while the fixes are still cheap and the agent doesn't have production traffic depending on the current design.

1. Can you reconstruct exactly what the model saw on any given call?

Before anything else: if you can't answer "what was in context on the call that produced this output," you can't debug context failures at all — you're stuck guessing between a model problem and a context problem every time something goes wrong. This is infrastructure, not an afterthought. Log the assembled context, not just the final response.

2. How many tools does the agent have registered, and how differentiated are they?

If the count is approaching twenty, or if any two tools could plausibly be confused for each other from their descriptions alone, this is a live liability, not a future one. Tool-selection accuracy degrades measurably past roughly ten tools, and every tool schema is standing overhead on every call regardless of whether it's used. Audit for near-duplicates and dead weight before shipping, and put a recurring version of this audit on the calendar — toolsets don't stay lean on their own.

3. Is retrieval returning the right document, or just a plausible one?

"The retrieval pipeline works" usually means "it returns something relevant" — a much weaker claim than "it returns the specific passage the model needs, ranked clearly above near-misses." A RAG system returning ten similar chunks when one specific chunk has the answer is diluting the model's attention on the way to using it. Test retrieval against queries where you know the exact correct passage, not just queries where any reasonable answer would look fine.

4. Does each retrieved chunk or tool result carry its own relevance signal?

The model shouldn't have to guess which of several similar-looking items in context is the one that actually matters for this call. If nothing in the context distinguishes the load-bearing item from the near-misses around it, that's a design gap, not something the model is expected to sort out through raw reasoning ability.

5. What happens to conversation history once it gets long?

If the answer is "nothing, it just keeps growing until we hit a token limit and something breaks," that's a compaction strategy that hasn't been designed yet — it's a future incident with a due date nobody's tracking. Decide explicitly what compresses safely (resolved sub-tasks, abandoned paths, conversational filler) and what has to survive verbatim (explicit constraints, exact numbers, decisions with downstream consequences) before the first long session forces the decision under production pressure.

6. Are stated constraints tracked as structured facts, or hoping to survive in the transcript?

A user preference or hard constraint stated once early in a conversation is exactly the kind of information a naive summarization pass tends to smooth away. If constraints aren't being extracted into something structured and persistent — rather than trusted to remain legible inside an increasingly compressed transcript — expect them to quietly stop being honored somewhere past turn thirty or forty.

7. Which long, noisy sub-tasks could be isolated instead of run inline?

Any step where the process required to get an answer is much longer than the answer itself — a debugging trail, a broad multi-file search, a research task synthesizing several sources — is a candidate for dispatching to a sub-agent, so the noise of getting there never enters the main agent's context at all. If every investigation currently happens inline, that's context bloat accumulating turn over turn that isolation would have prevented for free.

8. Do skills or reference material load on demand, or are they always present?

If the agent's skill or reference library is fully loaded into context on every call regardless of relevance, the cost of that library scales linearly with its size — and it'll be fine at five entries and a real problem at fifty. Check that only a lightweight index is always-present, with full content loading conditionally, and check that the one-line descriptions in that index are specific enough for the model to judge relevance without opening the full content.

9. Is memory a bigger context window, or is it actually structured?

If "remembering things about the user or the task" currently means "keep more history in a larger window," that's capacity, not memory — and it degrades the same way raw history always degrades once volume gets high enough. Structured, retrievable facts that get pulled in selectively per call outperform a bigger container holding everything, because the constraint that matters is attention under load, not how much the window can technically fit.

Running it as a practice, not a one-time audit

None of these nine questions are meant to be answered once, at launch, and filed away. Every one of them describes a condition that changes as an agent's responsibilities grow — a toolset that was lean at launch accretes tools over the following year; a memory system that was small enough to just work grows past the point where raw storage holds up; a skill library that started with five entries and tight descriptions grows to fifty with descriptions nobody revisited. The checklist isn't a launch gate. It's a set of questions worth re-asking on a cadence, because the answers degrade quietly, the same way the agent's performance does, without any single change looking like the cause.

That's really the throughline of this whole series: context engineering isn't a phase that happens before an agent ships and then is done. It's the ongoing discipline of making sure that everything the model sees, on every call, earned its place there — and re-checking that as the agent, and the demands on it, keep growing past whatever scale it was originally designed for.

None of these nine checks require exotic tooling to run. Most of them are answerable with a logging change, an afternoon spent reading through real production transcripts, and an honest count of what's actually registered as a tool or loaded as a skill. The teams that build the habit of running this list quarterly, rather than once at launch, are the ones whose agents stay reliable as they scale — not because their models are better, but because nothing in front of those models has been left to accumulate unexamined.

---

Part of the "Context Engineering" series on aiskill.market.