Observability for Agents Looks Nothing Like Observability for Services
Traditional APM tells you a request was slow. It can't tell you why an agent chose to call the refund tool instead of asking a human first.
A service either returns a 200 or it doesn't. Latency spikes, error rates climb, a graph turns red, someone gets paged. Decades of tooling — Datadog, New Relic, the entire APM industry — were built around this shape of failure: deterministic code taking a wrong branch, a dependency timing out, a queue backing up. You instrument the boundaries, watch the numbers, and when something breaks, the stack trace tells you exactly where.
Point that same tooling at an agent and it will tell you the request took 340ms and returned a 200. It will not tell you that the agent decided, on its own reasoning, to escalate a routine refund into a policy exception. It will not tell you that the tool call it made was technically valid but contextually wrong. The response was well-formed. The decision behind it was not observable at all.
This is the core mismatch: traditional monitoring captures whether a system executed correctly, but agents fail by executing correctly on the wrong judgment — and judgment doesn't show up in a status code.
The unit of failure changed
In a service, the unit of failure is a function call, a network hop, a database query. You can log each one, time each one, and reconstruct exactly what happened by reading the trace top to bottom. The system has no opinions — it does what the code says.
An agent's unit of failure is a decision: which tool to call, what to put in the arguments, whether to stop or keep going, how to interpret an ambiguous instruction. None of that is visible in a conventional trace unless you specifically capture it. A request/response log shows you the final tool call and its result. It doesn't show you the three tool calls the agent considered and rejected, the piece of context it weighted too heavily, or the instruction it silently reinterpreted. You get the output of the reasoning, not the reasoning.
That gap matters because most agent incidents aren't crashes. They're an agent doing exactly what it was designed to do, correctly, in a situation where that behavior was wrong. A perfectly executed action on a bad decision looks identical, from the outside, to a perfectly executed action on a good one — unless your observability captures the decision itself.
What agent observability actually has to record
Agent observability isn't APM with an LLM label slapped on it. It's a different data model, built around a few things services never needed to track:
- The full reasoning trace — not just "tool X was called," but what the model was given, what it inferred, and what it chose not to do.
- Tool-call arguments and their provenance — did this value come from user input, from a retrieved document, from the model's own inference? That distinction is the difference between an agent following instructions and an agent hallucinating them.
- Confidence and uncertainty signals — where they exist, whether the agent flagged its own output as low-confidence, and whether that flag was ever acted on.
- Multi-step context accumulation — an agent's fifth action is shaped by everything that happened in the first four. A trace that only shows step five in isolation is missing the causal chain.
This is why teams building for production track a distinct set of metrics that have no service-monitoring analogue at all: hallucination rate, policy-violation rate, escalation frequency, and audit-trail completeness. None of those exist in a conventional dashboard because none of those failure modes exist in conventional software. A service doesn't hallucinate a database row. An agent can hallucinate a customer's order history and act on it with complete internal confidence.
Guardrails without observability is flying blind, twice
Guardrails block a harmful action in the moment; observability is what tells you it happened at all, how often, and whether it's getting worse or better over time. Treat them as one system, not two features you bolt on separately. A guardrail that silently blocks a bad tool call and logs nothing gives you safety with zero visibility — you have no idea whether you're blocking three attempts a month or three hundred, whether the pattern is new, or whether your policy needs tightening before it causes real damage upstream of the block.
This is the piece that trips up teams moving fast: they ship the guardrail, watch it block a few things in testing, and call the safety work done. But a guardrail with no observability behind it is a alarm that goes off in an empty room. You built the tripwire. You didn't build the ability to notice it's tripping.
Why this shows up late, not early
Services fail loud and fast — a bad deploy usually pages someone within minutes. Agents fail quiet and slow. An agent with a subtly wrong instruction-following pattern doesn't crash; it just produces slightly-off outputs that look plausible enough to pass a glance, get accepted, and compound. Nobody notices for weeks, until the aggregate cost — refunds that shouldn't have gone out, tickets closed that shouldn't have been, data written that shouldn't have been touched — becomes too large to ignore.
By the time that surfaces through normal channels (a customer complaint, an audit, a finance reconciliation), you're not debugging a live incident anymore. You're doing forensics on weeks of decisions with no record of what the agent was actually reasoning through at the time. This is the scenario the five-step incident-response playbook exists to prevent — but step one of that playbook, detection, is entirely dependent on having captured the trace in the first place. No trace, no detection. No detection, no incident response — just a slow accumulation of damage that eventually becomes visible for reasons entirely outside your control.
Building the trace before you need it
The instinct is to add observability after the first bad incident. That's backwards, and it's backwards in a specific way: you can't reconstruct a decision trace retroactively. If you weren't capturing the reasoning, the tool arguments, and the context at the time, that information doesn't exist anywhere — it's not sitting in a log you forgot to check, it was simply never recorded.
Practically, this means treating trace capture as part of the agent's runtime, not an afterthought layered on top. Every tool call gets logged with its full argument set and the context that produced it. Every multi-step run gets a session ID that ties the steps together so you can replay the sequence, not just inspect isolated calls. Every point where the agent had a choice — continue, stop, escalate, ask for clarification — gets recorded as a decision point, even when the agent chose the boring option.
The payoff isn't just faster debugging. It's the ability to answer questions services never had to answer: is this agent's judgment getting better or worse over time? Is a particular class of task producing more escalations than it used to? Are we seeing a new pattern of prompt manipulation attempts that guardrails are catching but nobody's looking at? None of those questions have an answer in a system built to watch status codes. They only have an answer in a system built, from the start, to watch decisions.
Part of the "Running Agents in Production" series on aiskill.market.