Cheap Heuristics on 100% of Traffic, Expensive Judges on 10%

Somewhere in the life of every agent that goes from prototype to production, someone runs the math on what it would cost to judge every single request with an LLM call, and the number comes back wrong. If your agent is handling 50,000 requests a day and each judge call costs a few cents and adds a few hundred milliseconds of latency, judging everything means doubling your inference cost and adding a meaningful tax to every response, just to generate a score most of those requests will never need anyone to look at.

This is the moment teams either quietly stop running evals in production — reverting to offline test suites that never touch real traffic — or they discover the architecture that's become the default pattern across production agent systems in 2026: cheap, fast heuristics running on everything, and expensive LLM-as-judge scoring running on a sampled slice.

The two-tier shape of the pattern

The architecture has a simple shape once you see it, even though getting each tier right takes real work. Tier one is a fast, cheap evaluator — a distilled model, a rules-based heuristic, or a lightweight classifier — that runs on 100% of production requests and catches the failure modes that are both common and cheap to detect: malformed output, obvious refusals where none should occur, responses that fail a schema check, latency outliers, empty or truncated tool results treated as success. None of this requires the reasoning depth of a full LLM judge. It requires speed and complete coverage, because the entire value of tier one is that nothing slips through unmeasured.

Tier two is the expensive LLM-as-judge evaluation — potentially the full chain-of-thought, multi-dimensional, bias-mitigated setup described earlier in this series — running on a sampled 5 to 10% of requests, chosen either at random or weighted toward the request types most likely to carry subtle failures that tier one can't catch: long conversations, requests that triggered a tool error and recovered, requests flagged by a user as unhelpful, or requests from a cohort you're specifically watching after a recent change.

The two tiers aren't redundant with each other — they're catching different things. Tier one is a smoke detector: fast, cheap, covers the whole building, and only tells you something is wrong. Tier two is the fire investigator: expensive, slow, only dispatched to a fraction of alarms, but capable of explaining exactly what happened and why.

What belongs in the cheap tier

The discipline in building tier one is resisting the urge to make it smart. Its entire value proposition is that it's fast enough and cheap enough to run on everything, which means every check in it needs to be answerable without an LLM call, or with a call to a small, fast, purpose-distilled model rather than a frontier reasoning model.

Practical tier-one checks that scale to 100% of traffic: schema validation on structured outputs, regex or keyword checks for known-bad patterns (leaked system prompts, explicit policy violations, empty responses where content was expected), latency and token-count outlier detection, tool-call success/failure rates, and lightweight classifiers trained specifically to flag the two or three failure categories that show up most often in your own production history. That last one is worth dwelling on: a small classifier fine-tuned or prompted narrowly to detect one specific failure mode — say, "did this response commit to a factual claim about order status without actually calling the lookup tool" — can run cheaply at full volume and catch something a generic full judge would need much more careful prompting to catch reliably anyway.

What belongs in the expensive tier

Tier two is where the full machinery from earlier in this series belongs: the multi-dimensional rubric decomposition from "The Biases Baked Into Every AI Judge," trajectory grading from "Trajectory Evals: Grading the Path, Not Just the Answer," and potentially full agent-as-judge evaluation for the highest-stakes request categories. This tier can afford to be slow and expensive precisely because it's not running on everything — it's running on a sample deliberately chosen to be representative enough that its findings generalize back to the full traffic volume.

The sampling strategy itself deserves real thought rather than defaulting to pure randomness. A pure 10% random sample is a reasonable baseline, but a stratified sample — weighted toward request types where tier one has lower confidence, toward new feature paths that haven't accumulated much production history yet, or toward user segments where a regression would be costliest — gets more useful signal out of the same evaluation budget. The teams running this well tend to adjust their sampling weights the same way they'd adjust monitoring alert thresholds: based on where they've been burned before.

Where the two tiers meet: escalation

The sharpest version of this architecture doesn't just run two tiers in parallel and report both — it lets tier one escalate specific requests into tier two outside the random sample, when a cheap heuristic fires but isn't confident enough on its own to act. A request that trips a borderline latency threshold, or gets a marginal score from the fast classifier, gets automatically routed to the expensive judge for a real verdict, even if it wouldn't have been picked by the random sampling rate that request cycle. This turns the sampled tier into something closer to a targeted second opinion than a pure statistical sample, and it's usually the highest-leverage addition a team makes to this architecture once the basic two-tier version is running.

Why this isn't just a cost optimization

It's tempting to file this pattern under "cost engineering" and move on, but the sampling architecture solves a second problem that pure cost-cutting wouldn't: latency. A full LLM-as-judge call in the critical path of every user-facing request adds real, user-perceptible delay, and for agents where response time matters — which is most of them — that tax is not something teams can absorb just because their budget technically allows it. Running the expensive judge asynchronously, on a sample, off the critical path, is what makes it possible to keep the deep evaluation without making every user wait for it.

This is also the architectural bridge into the next piece in this series, "Turning Evals Into Guardrails That Run at Inference Time," which covers what happens when some subset of the cheap tier's checks move from after the response is generated to before it's returned to the user — blocking or rewriting bad outputs in real time rather than just measuring them after the fact. The two-tier sampling pattern and inline guardrails aren't competing approaches; they're adjacent layers of the same underlying idea, which is that not every eval needs to run at the same cost, speed, or point in the request lifecycle. Matching each check to the tier where its cost is justified by what it catches is the actual skill here — not running more evals, and not running fewer, but running the right ones at the right price for what they're each responsible for catching.

Part of the "Evals for Agents" series on aiskill.market.