Evals for Agents
10 articles in series
Ten essays on how builders actually test agentic systems in 2026 — the biases baked into every LLM judge, trajectory evals, agent-as-judge, cheap heuristics at 100% traffic vs expensive judges at 10%, and a starter eval suite for your first production agent.
An Eval Suite Starter Kit for Your First Production Agent
A concrete, ordered starting eval stack for a small team shipping their first production agent — what to build first, and what to skip until it hurts.
What Correlating Judge Scores to Human Ratings Actually Looks Like
The 0.85 correlation threshold everyone cites is easy to state and tedious to actually reach. Here's the real, unglamorous process of validating a judge.
Turning Evals Into Guardrails That Run at Inference Time
Offline eval suites catch what already shipped. Runtime guardrails catch bad outputs before a user ever sees them. Here's how the two connect.
Why Eval-Driven Development Is Replacing Vibe-Checking Outputs
Reading five outputs and deciding a prompt 'feels better' doesn't scale past the first week. Eval-driven development writes the test before the fix.
Agent-as-Judge: Letting an Agent Grade Another Agent's Work
Agent-as-judge gives evaluators tools, memory, and multi-step reasoning to grade full trajectories — but it inherits every bias a simple judge has, plus new ones.
Cheap Heuristics on 100% of Traffic, Expensive Judges on 10%
Running LLM-as-judge on every production request doesn't scale on cost or latency. Here's the sampling architecture teams actually run instead.
Building a Golden Dataset When You Don't Have Real Traffic Yet
You can't mine production logs for eval cases before you have users. Here's how solo builders and small teams bootstrap a golden dataset from zero.
Trajectory Evals: Grading the Path, Not Just the Answer
An agent that stumbles into the right answer through wasted tool calls isn't working. Trajectory evals grade the reasoning path, not just the output.
The Biases Baked Into Every AI Judge (and How to Cancel Them Out)
Position bias, verbosity bias, and self-preference bias distort every LLM-as-judge score. Here's what each looks like and the concrete fixes that work.
LLM-as-Judge Was the Start, Not the Finish
LLM-as-judge solved the eval bottleneck in 2024. By 2026 it's table stakes — here's what production teams layer on top of it now.