Agent-as-Judge: Letting an Agent Grade Another Agent's Work

A single-pass LLM judge reading a trajectory has a real limitation that no amount of prompt engineering fixes: it can only reason about what's already written in the trace. If an agent's trajectory shows a tool call that returned an ambiguous result, the judge can speculate about whether the agent's next move was justified, but it can't go verify the ambiguity itself — it can't call the tool again with different parameters, look up the actual current state of the system, or check whether a claim the agent made about "the only available option" was actually true at the time. It's grading from a transcript, the way a reviewer grades a chess game from the move list alone, without a board in front of them.

Agent-as-judge removes that limitation by giving the evaluator the same kind of capability the thing it's evaluating has: multi-step reasoning, state management, and tool use, applied to the job of judging an entire trajectory rather than a single output. Instead of reading a static trace and guessing, an agent-as-judge can go investigate — query the actual system state, re-run a lookup to check whether the agent's tool call returned accurate information, or walk through a multi-step trajectory the way a senior engineer would review a colleague's incident postmortem, asking follow-up questions as they go rather than accepting the write-up at face value.

What agent-as-judge can do that a static judge can't

The clearest case for agent-as-judge shows up exactly where trajectory evaluation gets hardest: verifying claims the judged agent made about the world. If an evaluated agent's final answer says "I checked and there's no earlier flight available," a static LLM judge reading the trace can only check whether the trace shows a flight search — it has no way to independently confirm the claim was true. An agent-as-judge with access to the same or an equivalent search tool can actually run the check itself and compare.

This extends naturally to longer, more consequential trajectories. A coding agent that made a series of edits across multiple files, ran tests, and reported success is exactly the kind of case where a static judge reading a diff and a test-output log is working from secondhand evidence. An agent-as-judge with repository access can check out the change, re-run the test suite independently, and confirm the claim rather than trust the reported output — closing the same gap that trajectory evals closed relative to final-answer judging, one layer deeper.

This is a meaningful capability jump from the trajectory grading covered in "Trajectory Evals: Grading the Path, Not Just the Answer," which is largely about reading a trace more carefully. Agent-as-judge is about not having to take the trace's word for it.

The recursive bias problem

Here's the part that makes agent-as-judge genuinely harder than it first appears, and not just a strictly-better upgrade you should reach for by default: an agent-as-judge is itself an agent, subject to every failure mode covered in "The Biases Baked Into Every AI Judge," plus a new one that only shows up once the judge starts taking its own actions — it can develop the same blind spots as the agent it's evaluating, for the same underlying reasons.

Concretely: if the evaluated agent tends to under-verify tool results before acting on them, and the judge agent is built on the same underlying model family with the same tendencies, the judge may not flag that under-verification as a problem, because from the judge's own operating pattern, that level of verification looks normal. A judge that shares its evaluated agent's blind spots isn't a neutral referee — it's closer to a peer reviewing a peer's work in a shared house style, where the things both of them do without thinking don't register as issues to either one.

This is self-preference bias from the earlier piece, but operating at the level of behavior rather than just output style. A same-family judge doesn't just prefer the evaluated agent's phrasing — it can share its actual reasoning shortcuts, which is a much harder thing to detect, because the judge's own investigative actions will tend to validate rather than question the exact category of shortcut it's also prone to taking.

Cost and complexity are not incidental

A second, more mundane reason agent-as-judge isn't a default upgrade: it's expensive and slow in a way that compounds the sampling-architecture problem from "Cheap Heuristics on 100% of Traffic, Expensive Judges on 10%." A static LLM judge is one API call. An agent-as-judge is a full agentic run — potentially multiple tool calls, multiple reasoning steps, its own retries and error handling — which means it inherits all the cost and latency characteristics of the very systems it's evaluating. Running agent-as-judge on even a 10% sample of high-volume production traffic can be a meaningfully larger expense than running a static judge on the same sample, and the sampling rate for agent-as-judge in practice tends to be a fraction of the fraction — reserved for the highest-stakes trajectory categories rather than applied broadly.

Where agent-as-judge actually earns its cost

Given the added complexity and expense, agent-as-judge makes the most sense in a narrower set of situations than "grading agent work" broadly:

High-stakes, low-volume trajectories — financial transactions, irreversible write actions, anything where a single missed failure is expensive enough to justify a deep, independent-verification review rather than a fast read of the trace.

Trajectories with externally verifiable claims — cases where the evaluated agent asserts something about the state of the world that a tool-using judge can actually go check, rather than purely subjective judgments of tone or helpfulness where independent verification adds nothing.

Incident review and postmortem analysis — after something has already gone wrong, using an agent-as-judge to reconstruct exactly where a trajectory diverged from correct behavior is a genuinely strong fit, because you're not optimizing for cost and speed at that point, you're optimizing for a thorough answer.

Cross-checking a static judge's most surprising verdicts — rather than replacing static judging wholesale, routing only the cases where a static judge's score is unexpectedly high or low to an agent-as-judge for a second, deeper opinion, which gets most of the value at a fraction of the cost of running agent-as-judge on everything.

The honest framing

Agent-as-judge is a real capability, not hype dressed up in agentic language — it closes a genuine gap that static trajectory judging can't close on its own, the gap between reading a claim and verifying it. But it's not a strictly better replacement for the judging patterns earlier in this series, and treating it as one is how teams end up with an expensive, slow evaluation layer that inherited the exact blind spots it was supposed to catch. The right mental model is a specialist you call in for specific cases, not a general upgrade you apply everywhere — and like every other judge in this series, its verdicts are only as trustworthy as the last time someone checked them against a human's.

Part of the "Evals for Agents" series on aiskill.market.