The Biases Baked Into Every AI Judge (and How to Cancel Them Out)

A team I'll describe generically — because this exact story has played out at enough companies that naming one would be misleading — shipped a prompt change that made their support agent's responses noticeably more verbose. Eval scores went up 8%. They shipped it. User satisfaction, measured separately through post-interaction surveys, didn't move. The judge hadn't detected an improvement. It had detected more words.

This is verbosity bias, and it's one of three well-documented distortions that live inside every LLM-as-judge setup, whether or not the team running it knows to look for them. If you're using an LLM to grade another LLM's output — which, per the previous piece in this series, "LLM-as-Judge Was the Start, Not the Finish," is still the right foundation to build on — you need to know exactly how that judge lies to you before you can trust what it says.

Position bias: the judge has a favorite seat

When you show a judge model two responses side by side and ask it to pick the better one, the response's position in the prompt — first or second — measurably affects the outcome, independent of quality. Some judge models favor whichever response appears first; others favor whichever appears last. The direction varies by model and prompt template, but the existence of the effect doesn't.

This matters enormously for anything doing pairwise comparison: A/B testing two prompt variants, comparing a new model version against the old one, or running a champion/challenger setup for a skill or agent behavior. If you always put the new variant in the same slot, you're not measuring quality — you're measuring a seating preference the judge didn't know it had.

The fix is mechanical, not philosophical: rotate response positions across evaluation runs so that position bias cancels out in aggregate rather than silently favoring whichever variant you happened to list first. Run each comparison twice, once with A first and once with B first, and average. It costs you a second judge call per comparison. It's worth every token.

Verbosity bias: longer reads as better, even when it isn't

This is the bias that bit the team in the opening story, and it's arguably the most consequential one for anyone doing prompt iteration, because prompt changes that add instructions ("be thorough," "explain your reasoning," "provide context") reliably produce longer outputs — and longer outputs reliably score higher with a holistic judge, independent of whether the added length carries any additional value.

The mechanism is intuitive once you see it: a judge asked "which response is better" without a tightly specified rubric will lean on surface cues, and length is one of the strongest surface cues a model can latch onto, because verbose answers look thorough even when they're padded with hedges and restatement. A one-sentence rejection of a bad idea and a five-paragraph rejection of the same bad idea contain the same amount of actual information, but only one of them reads as "the judge did real work here."

Decomposing the rubric is the direct countermeasure, and it's a big enough shift that it's worth stating as its own principle: break a holistic quality rubric into discrete, single-criterion checks — factual accuracy, task completion, tone, conciseness — rather than asking for one score, because a judge that has to explicitly grade conciseness can no longer let length quietly substitute for quality everywhere else. A single "how good is this response, 1 to 10" prompt gives verbosity bias nowhere to hide. Five separate prompts, each scoring one dimension, box it into the one dimension where it's actually relevant.

Self-preference bias: the judge likes its own family

The third bias is the one that surprises people most on first hearing it: a judge model scores outputs produced by its own model family more favorably than outputs of comparable quality produced by a different model. Ask a judge from one lab to compare a response from its own family against a response from a competing lab's model, and even when a blinded human rater would call the two responses equivalent, the judge tends to prefer its own family's output.

This isn't a conspiracy or deliberate favoritism baked in on purpose — the leading theory is that models share stylistic and structural conventions within a family (how they organize an answer, what kind of hedging language they use, how they format lists), and a judge from that family recognizes those conventions as "the way a good answer looks" simply because that's the distribution it was trained on. The practical consequence is the same regardless of cause: any eval setup comparing outputs across model providers needs to treat same-family judging as a confound, not a footnote.

If you're deciding whether to switch your agent's underlying model, or you're running a multi-model ensemble and using a judge to pick the best response among them, using a judge from the same family as one of the candidates tilts the field before the comparison starts. The workaround is either to use a judge from a third, unrelated model family for cross-provider comparisons, or to run the comparison with multiple judges from different families and look for agreement rather than trusting any single judge's verdict.

Chain-of-thought judging: the fix that also gives you a debug trail

All three biases above share a common root: a judge asked to produce a single score with no explanation has nowhere to put its reasoning except into an opaque number. Forcing the judge to explain itself before scoring — chain-of-thought judging — closes some of that gap directly. Having the judge articulate its reasoning step by step before assigning a score has been shown to improve reliability by 10 to 15%, and it produces something a holistic score never gives you: a debuggable trail. When a chain-of-thought judge's stated reasoning contradicts its final score, that mismatch is itself a signal something's wrong with either the rubric or the judge prompt — a failure mode a bare numeric score simply cannot surface, because there's no reasoning to check it against.

Validating against something you trust

None of these mitigations are enough on their own to make a judge trustworthy in the abstract — they need to be checked against ground truth, which for eval purposes means human ratings. The discipline that actually closes the loop is validating judge output against human ratings until the correlation exceeds 0.85, and re-running that validation whenever the judge prompt, rubric, or underlying model changes — a process detailed in "What Correlating Judge Scores to Human Ratings Actually Looks Like," later in this series. Position rotation, rubric decomposition, and chain-of-thought reasoning are the tools that get a judge into the range where that correlation is achievable. They're not a substitute for actually measuring it.

The discipline is boring, which is why it works

None of this is exotic engineering. Rotating positions is a loop. Decomposing a rubric is a prompt-writing exercise. Chain-of-thought judging is one more instruction appended to a prompt template. What makes these mitigations easy to skip is that skipping them doesn't produce an obvious error — your eval pipeline runs fine, your dashboard updates, your numbers look plausible. The failure is silent: a biased judge doesn't crash, it just quietly tells you the wrong thing is working. The teams that catch this early are the ones who built the habit of asking, every time a score moves, whether the underlying behavior moved or just the judge's blind spot did.

Part of the "Evals for Agents" series on aiskill.market.