Can AI Judge AI?
What LLM-as-judge gets right, where it quietly fails, and how to calibrate it before you trust it.
LLM-as-judge is the most seductive shortcut in evaluation. Point a capable model at two responses, ask which is better, and you get a number in seconds. The number feels objective. It is not.
What it gets right
For coarse, high-contrast comparisons — a coherent answer versus a broken one — judges agree with humans often enough to be useful. They are cheap, fast, and tireless, which makes them ideal for the first pass of a large sweep.
Where it quietly fails
The failures are rarely loud. Judges reward length, confidence, and formatting. They inherit the position bias of their prompt. They score fluent nonsense above terse truth. None of this shows up unless you go looking for it.
Calibrating before you trust it
Before a judge gates anything that matters, anchor it: a small human-labeled set, agreement measured, biases probed with swapped positions and padded answers. A judge you have calibrated is an instrument. A judge you have not is a rumor.