What Correlating Judge Scores to Human Ratings Actually Looks Like

The number gets cited constantly in eval discussions: validate your judge against human ratings until the correlation exceeds 0.85. It's stated as a target, almost like a hyperparameter you tune until the dashboard shows a green checkmark. What that framing skips over entirely is that reaching 0.85 correlation is not a number you optimize toward by adjusting a threshold — it's the output of a genuinely tedious human process, and most of the value in doing it comes from the disagreements you surface along the way, not from the final correlation coefficient itself.

Here's what the process actually looks like when a team does it properly, rather than skipping straight to trusting a judge because its prompt looks reasonable.

Step one: humans rate the same cases the judge will rate

You start with a set of cases — ideally the golden dataset covered in "Building a Golden Dataset When You Don't Have Real Traffic Yet," since it's already the set of cases you've decided matter — and you have real humans rate the outputs using the exact same rubric the judge will use. Not a looser, more intuitive version of the rubric. The same one, criterion by criterion, so the two ratings are actually comparable rather than measuring different things dressed up as the same score.

This step alone is where most teams underinvest. A single person rating fifty cases in twenty minutes produces ratings, but not reliable ones — rater fatigue, inconsistent standards across the session, and the rater's own biases (their own version of position and verbosity bias, just human-shaped) all creep in. The teams that get a validation process they can actually trust use at least two independent human raters per case, rating separately without seeing each other's scores, specifically so the next step is possible.

Step two: check whether the humans agree with each other before checking whether the judge agrees with them

This is the step that gets skipped constantly, and skipping it invalidates everything downstream. If two human raters scoring the same case under the same rubric only agree with each other 60% of the time, then a judge that also only agrees with the humans 60% of the time isn't necessarily a bad judge — it might be performing at the actual ceiling of what's achievable, because the rubric itself is ambiguous enough that reasonable humans read it differently.

Inter-rater agreement between the humans is the ceiling your judge is realistically working toward, not a footnote — a judge can't be meaningfully more consistent with human judgment than humans are with each other, and finding out your raters disagree with each other more than expected is itself a critical finding, because it usually means your rubric needs to be rewritten before the judge validation means anything at all. Teams that discover low human-human agreement almost always trace it to a rubric criterion that sounds precise but isn't — "helpfulness" or "quality" without a concrete operational definition of what separates a 3 from a 4. The decomposition principle from "The Biases Baked Into Every AI Judge" — breaking a holistic score into discrete, single-criterion checks — pays off again here, because narrow, concrete criteria are exactly the ones two humans are most likely to agree on independently.

Step three: run the judge on the same cases and compute the actual correlation

Only once you trust the human ratings do you run the judge — using the identical rubric, identical cases, identical scoring scale — and compute the correlation between the judge's scores and the (averaged, or reconciled) human scores. This is a genuinely mechanical step, and it's the only part of the whole process that resembles the "tune until you hit 0.85" framing the number gets discussed with. Everything before it is the actual work.

When the correlation comes in below target, the productive next move isn't to keep re-running the same judge prompt hoping for a better roll — it's to go read the specific cases where the judge and the humans diverged most sharply, and look for a pattern. Sometimes the pattern is a bias covered earlier in this series: the judge is quietly rewarding length, or favoring a particular response position, or showing self-preference against a competing model's output. Sometimes the pattern is that the rubric criterion itself is one the judge structurally can't evaluate well from the information it's given — asking a judge to assess "did this response show appropriate empathy" from text alone, when human raters were implicitly drawing on tone cues the judge's rubric never asked it to attend to explicitly.

Step four: re-validate every time something changes

The correlation you measure is only true for the exact judge prompt, rubric, and underlying model you measured it with. Change any of the three — a prompt tweak to reduce verbosity bias, a rubric criterion rewritten after step two revealed ambiguity, or simply an upgrade to a newer version of the underlying judge model — and the correlation you validated is no longer known to hold. It might still be fine. It might have quietly dropped below the threshold and nobody would notice, because most teams treat validation as a one-time gate passed at launch rather than a recurring check.

This is the same discipline problem that runs through the rest of this series: validation that happens once and is then trusted forever is functionally the same mistake as vibe-checking a prompt change and never writing it down as a permanent test, the exact failure mode covered in "Why Eval-Driven Development Is Replacing Vibe-Checking Outputs." The fix is the same instinct applied here: treat judge-prompt changes the way you'd treat any other change to production logic, with a re-validation step baked into the process rather than left to memory.

Why this is worth the tedium

None of this is glamorous work. It's humans reading outputs, disagreeing with each other, rewriting a rubric criterion for the third time because "clarity" turned out to mean five different things to five different raters, and running the same correlation calculation again after every adjustment. It's the least automatable part of an otherwise heavily automated eval pipeline, and that's exactly why it's the part most likely to get skipped under deadline pressure.

Skipping it doesn't remove the risk — it just moves the discovery of a miscalibrated judge from a controlled validation exercise to a production incident, where the first sign of the problem is a pattern of user complaints that don't match what the eval dashboard has been reporting for months. The 0.85 correlation figure is a reasonable target precisely because it's demanding enough to force this process to happen properly rather than being satisfied by a token pass at it. Every other piece of eval infrastructure in this series — the bias mitigations, the trajectory grading, the sampling architecture, the runtime guardrails — is only as trustworthy as the judge underneath it, and this tedious, unglamorous validation loop is the only thing that actually earns that trust rather than assuming it.

Part of the "Evals for Agents" series on aiskill.market.