LLM-as-Judge Was the Start, Not the Finish

Ask ten engineering teams building AI agents how they know their agent is any good, and eight of them will describe some version of the same setup: a held-out set of test cases, an LLM prompted to score the outputs, and a dashboard that turns those scores into a number that goes up and to the right. This is LLM-as-judge, and for a while it felt like the answer to a problem that had no answer — how do you grade free-form natural language output at scale without hiring an army of human raters.

It still works. It's still the right first move for almost anyone starting from zero. But treating it as the finished solution rather than the entry point is where most eval programs quietly stall out. The single-number judge score becomes the thing everyone optimizes against, the dashboard everyone stares at, and the reason nobody notices that the agent's tool-calling logic has been silently degrading for three weeks because the final answer still reads fine.

Why the single score always felt like enough

The appeal of LLM-as-judge was never subtle. Before it, evaluating a generative system meant either writing brittle string-matching tests that broke the moment your prompt changed a single word, or paying humans to read outputs one at a time — slow, expensive, and impossible to run on every commit. A judge model that could read a response and a rubric and return a 1-to-5 score solved both problems at once. You could run it in CI. You could run it on a thousand examples overnight. You could watch the average trend over time and treat regressions as build failures.

That's a genuinely good pattern, and nothing in this series argues you should stop doing it. The problem is what happens when it's the only pattern. A single scalar collapses everything an agent does — reasoning quality, tool selection, factual grounding, tone, safety — into one number, and one number can't tell you which of those five things broke. Worse, the judge itself carries its own failure modes, which is the subject of the next piece in this series, "The Biases Baked Into Every AI Judge": judges systematically favor longer answers, favor answers in certain positions, and favor answers that sound like their own model family wrote them.

What changed between 2024 and 2026

Two years of production experience across thousands of agent deployments surfaced a consistent pattern: teams that treated LLM-as-judge as their entire eval strategy hit a ceiling where their agents kept passing evals while user complaints kept climbing. The gap wasn't the judge lying — it was the judge answering a narrower question than the one the team thought it was asking. "Is this final response good?" is not the same question as "did this agent take a sane path to get here, use its tools correctly, and avoid a costly detour along the way?"

That distinction is why 2026's consensus view is that LLM-as-judge, while still useful, is no longer sufficient on its own. What replaced the single-judge-score-as-north-star model is a stack with three layers, each covered later in this series:

• Multi-dimensional metrics instead of one holistic score — separate checks for factual accuracy, tool-use correctness, tone, and safety, each scored independently rather than blended into a single number a human has to reverse-engineer.
• Observability as the foundation — trace-level visibility into what the agent actually did (which tools it called, in what order, with what arguments, and what came back) rather than only what it said at the end.
• Evals embedded directly into the runtime as guardrails, not just an offline test suite that runs before deploy and never touches production traffic again.

The trajectory problem

Here's the concrete failure that pushed teams toward this shift. An agent tasked with looking up a customer's order status calls the wrong API endpoint, gets an error, silently retries with a different endpoint, gets a partial result, and then — because the underlying model is good at sounding confident — writes a final answer that reads as clean and correct. A judge scoring only the final response gives this a 5. A human reading the trace would flag it immediately: the agent got lucky, not right, and next time the fallback might not exist.

This is the case for what the industry now calls trajectory evaluation — grading the path an agent took, not just where it landed. It's a big enough shift in how teams think about evals that it gets its own full treatment later in this series, in "Trajectory Evals: Grading the Path, Not Just the Answer." For now, the point is narrower: a judge that never sees the trajectory is structurally blind to an entire category of agent failure, no matter how well-calibrated its prompt is.

Agent-as-judge, and why it's not a free upgrade

The natural next move, once you accept that trajectories matter, is to stop asking a single LLM call to judge the trajectory and instead give the judge itself the tools to investigate — let it call functions, check intermediate state, and reason step by step before it scores. This is agent-as-judge: a full agentic system, capable of multi-step reasoning and tool use, evaluating another agent's entire reasoning trajectory rather than just its output.

It's a real capability upgrade, and it's covered in depth in "Agent-as-Judge: Letting an Agent Grade Another Agent's Work." But it inherits every bias problem a simple judge has, plus a new one: now you have an agent grading an agent, and if both share failure modes — say, both underweight the cost of an unnecessary API call — the judge won't catch what it's also prone to doing itself.

What this series actually covers

The honest starting point for anyone building agent evals in 2026 is not "should I use LLM-as-judge" — that question is settled, and the answer is yes, as one layer among several. The real questions are operational: how do you decompose a rubric so a judge isn't guessing at a holistic vibe, how do you keep judge costs sane when you're running millions of production requests, how do you build a golden dataset before you have real traffic to mine, and how do you verify that your judge's opinion actually correlates with what a human would say.

This series works through each of those questions with the specificity they deserve. Bias and its mitigations. Trajectory grading. Golden datasets without production data. The 100%-cheap / 10%-expensive sampling architecture that's become the default production pattern. Agent-as-judge and its recursive risks. The shift from vibe-checking to eval-driven development. Runtime guardrails. Correlation validation against human ratings. And finally, a concrete starter kit for a team shipping its first production agent.

None of it works if you start from the premise that a good judge prompt is the whole job. It's the first ten percent. The other ninety percent is architecture — how evals sit relative to your agent's execution, how much they cost to run, and how much you actually trust the number they hand you at 2am when a deploy just went out and something feels off.

Part of the "Evals for Agents" series on aiskill.market.