Trajectory Evals: Grading the Path, Not Just the Answer

Picture an agent with access to a database write tool, tasked with updating a customer's shipping address. It reads the request, calls the wrong table first, catches the resulting error, corrects course, calls the right table, and successfully updates the address. The final answer to the user — "Your address has been updated" — is completely correct. A judge scoring only that final message gives it full marks.

But something happened in between that a final-answer judge never sees: the agent's first call touched a table it shouldn't have queried at all, and if that first call had been a write instead of a read, the mistake would have been a real incident, not a recoverable stumble. The agent didn't fail this time. It also didn't demonstrate that it wouldn't fail next time, under slightly different conditions, with a slightly different tool available. That gap — between "produced a correct output" and "took a trustworthy path to it" — is exactly what trajectory evaluation exists to close.

Why final-answer grading was fine for chatbots and isn't fine for agents

The instinct to grade only the final output comes from a reasonable place: it's how LLM-as-judge evaluation started, back when the systems being evaluated were single-turn chatbots with no tools and no state. If a model just generates text in response to text, the output is the entire behavior worth grading — there's no path, only a destination.

Agents broke that assumption the moment they started calling tools, maintaining state across turns, and making sequences of decisions where each decision opens or forecloses options for the next one. An agent's trajectory — the ordered sequence of tool calls, intermediate reasoning, retries, and state changes it produces on the way to an answer — carries information a final answer can't. Two agents can arrive at the same correct answer through radically different paths, and only one of those paths is the one you want running in production at scale, because the other one got there by luck, by an unnecessary detour that happened not to cause damage, or by burning three times the tool calls and cost for the same result.

This is the throughline from "LLM-as-Judge Was the Start, Not the Finish": a judge that only ever sees the final message is structurally blind to an entire category of failure. Trajectory evals are how you open that blind spot up.

What a trajectory eval actually checks

A trajectory eval isn't one thing — it's a family of checks applied to the full execution trace rather than the final string. In practice, the useful dimensions tend to cluster into a few buckets:

Tool selection correctness. Did the agent choose the right tool for the sub-task at each step, or did it reach for something adjacent that happened to work? An agent that uses a broad search tool when a direct lookup was available isn't wrong in the sense of producing a bad answer, but it's burning latency and cost that a well-calibrated agent wouldn't.

Necessity and redundancy. Did every tool call in the trace need to happen? Agents under light supervision develop a habit of re-verifying things they already confirmed two steps earlier, or calling a tool speculatively "just in case." Each redundant call is invisible in the final answer and fully visible in the trajectory.

Recovery quality. When something did go wrong — an API error, an empty search result, a malformed argument — how did the agent respond? Good recovery looks like a targeted retry with corrected parameters. Bad recovery looks like blind retries of the exact same failing call, or worse, silently proceeding as if the failed step had succeeded.

State and side-effect safety. For agents with write access — to a database, a calendar, a customer record, a codebase — the trajectory is the only place you can check whether a destructive or irreversible action was taken unnecessarily, taken with the wrong parameters and then corrected, or taken in an order that would have caused a problem under different timing.

None of these show up in a final-answer judge prompt, because a final-answer judge never receives the trace — only the output. Trajectory evaluation requires your eval pipeline to have access to the full execution log, which is exactly why observability is now treated as foundational infrastructure for agent evals rather than a nice-to-have added after the fact.

The observability dependency nobody budgets for

Here's the part that catches teams off guard: you cannot run a trajectory eval on an agent whose tool calls, intermediate reasoning, and state changes aren't being logged in a structured, queryable way. If your only record of what an agent did is the final message it produced, you have no trajectory to grade — you've thrown the information away before the eval pipeline ever gets a chance to look at it.

This is why the 2026 consensus treats observability as a foundation layer, not an add-on. Every tool call, its arguments, its return value, every intermediate reasoning step, and every state mutation needs to be captured in a trace format the eval system can consume. Teams that bolt trajectory evaluation onto an agent built without this in mind usually discover they need to instrument the agent first — which is a real engineering task, not a config flag — before they can write a single trajectory-grading rubric.

Grading a trajectory without drowning in complexity

The temptation, once you see how much information lives in a trace, is to try to grade all of it at once with one enormous rubric. That approach reproduces the exact problem covered in "The Biases Baked Into Every AI Judge," at a larger scale — a single holistic score over a long, structured trace gives a judge model even more surface area to latch onto superficial cues like trace length rather than actual correctness.

The pattern that works better is the same decomposition principle applied to trajectories instead of single responses: separate checks for tool selection, redundancy, recovery, and safety, each scored independently, rather than one "was this a good trajectory" prompt fed the entire trace. It's more setup work upfront, and it pays for itself the first time a trajectory score drops and you can immediately tell whether the cause was a new redundant retry pattern or an actual safety violation, instead of staring at one number wondering which of a dozen possible things went wrong.

Where this leads: an agent that can read the trace itself

Once you're grading full trajectories rather than final outputs, the natural next question is whether the grading itself should be done by something more capable than a single-pass LLM call — something that can investigate a long trace, check intermediate state, and reason across multiple steps the way the agent it's grading did. That's agent-as-judge, and it's substantial enough to warrant its own treatment in "Agent-as-Judge: Letting an Agent Grade Another Agent's Work," later in this series.

For now, the practical takeaway is narrower and more urgent: if your eval suite still consists entirely of final-answer scoring, you're measuring whether your agent occasionally gets lucky, not whether it's reliable. The path an agent takes to an answer is not incidental detail — for any agent with tool access, it is often the single largest source of production risk that a final-answer judge cannot see at all. Instrument the trace, decompose the rubric, and grade the path before you grade the destination.

---

Part of the "Evals for Agents" series on aiskill.market.