An LLM judge is only useful if its verdicts match what a careful person would say about the same output. The rubrics that get there are narrow: one criterion per call, a written definition of pass and fail, a way to say "unknown", and a check against human labels using a statistic that corrects for chance, not raw percent agreement.
This post covers what the research says about how far LLM judges can be trusted, the biases to design around, and working code for the rubric, the position-swap check and the agreement math.
How well do LLM judges agree with humans?
Well enough to be useful, and not well enough to skip checking.
The best-known study is "Judging LLM-as-a-Judge" by Zheng and colleagues, the paper behind MT-Bench. When they compared GPT-4's pairwise verdicts with human experts and dropped ties, the two agreed 85 percent of the time on first-turn questions. Humans agreed with each other 81 percent of the time on the same setup. When ties were counted, both numbers fell, to 66 percent for GPT-4 against humans and 63 percent between humans.
G-Eval, a scoring method from Liu and colleagues, reports a Spearman correlation of 0.514 between GPT-4 scores and human ratings of news summaries. That beat every other automatic metric the paper compared it with, and it is still a moderate correlation.
Two things follow. First, a judge can be about as consistent with people as people are with each other, which is a fair bar. Second, both papers tested 2023 models on their own tasks. Your judge, on your outputs, has its own agreement rate, and you only learn it by measuring.
The biases to design around
The MT-Bench paper names four ways judges go wrong. Each one suggests a fix.
- Position bias. In pairwise comparisons, judges favor one slot. With the paper's default prompt, GPT-4 gave the same verdict after the two answers were swapped in 65 percent of cases, and Claude-v1 in 23.8 percent. The fix is to ask twice with the order swapped.
- Verbosity bias. The authors padded answers by repeating their content as a longer list. That fooled Claude-v1 and GPT-3.5 in 91.3 percent of cases and GPT-4 in 8.7 percent. The fix is a rubric that says length is not a criterion, and test cases that check it.
- Self-enhancement. GPT-4 gave its own answers a 10 percent higher win rate, and Claude-v1 its own a 25 percent higher one. The authors say their data was too limited to confirm the bias. G-Eval saw something similar: its GPT-4 judge always rated GPT-3.5 summaries above human-written ones, even when human raters preferred the human ones. Anthropic's evaluation docs give the general fix, which is to grade with a different model from the one that produced the output.
- Weak grading of math and reasoning. Judges were misled by wrong answers even on basic math. Giving the judge a reference solution cut the failure rate on the paper's math questions from 70 percent to 15 percent.
Pass/fail or a 1 to 5 scale?
For most agent checks, use pass/fail per criterion.
Hamel Husain, in his guide to LLM judges, argues against uncalibrated 1 to 5 scales across several dimensions, where the gap between scores is unclear and subjective. His point about acting on them is blunter: people do not know what to do with a 3 or a 4.
Anthropic's documentation asks for the same thing in different words: have the judge output only "correct" or "incorrect", or a score on a defined 1 to 5 scale, because purely qualitative feedback is hard to assess at scale.
If you do need a scale, for example to rank summaries, G-Eval shows one way to make it smoother. Instead of taking the single digit the model prints, it weights each possible score by the model's probability of giving it:
score = p(1) × 1 + p(2) × 2 + ... + p(5) × 5
The paper does this because one digit tends to dominate and models print whole numbers even when asked for decimals, which produces many ties. In its ablation, dropping the probabilities lowered the average correlation only slightly, from 0.514 to 0.502, so treat this as a refinement, not the main lever.
How to write the rubric
Anthropic's guidance on agent evals is to give each dimension of a task its own clear rubric and grade each one with a separate judge call, rather than one call that grades everything. It also recommends giving the judge a way out, such as returning "Unknown" when it lacks the information to decide. Their developer docs add that the judge should reason before it gives a score.
Put together:
import re
from typing import Callable
JUDGE_PROMPT = """You are grading one response from a customer support agent.
Criterion: {name}
Pass if: {pass_if}
Fail if: {fail_if}
<conversation>
{conversation}
</conversation>
<response>
{response}
</response>
Think it through in <reasoning> tags first. Then give exactly one verdict in
<result> tags: PASS, FAIL, or UNKNOWN if the conversation does not give you
enough information to decide."""
RUBRIC = [
{"name": "Resolves the request",
"pass_if": "the response answers the customer's actual question or completes the action they asked for",
"fail_if": "it answers a different question, asks for information already given, or stops before the task is done"},
{"name": "States only supported facts",
"pass_if": "every policy, price or date in the response appears in the conversation or the tool results",
"fail_if": "it states a policy, price or date that the conversation does not support"},
]
def grade(call_model: Callable[[str], str], conversation: str, response: str) -> dict:
"""Grade each criterion in its own call, so one verdict can't color another."""
verdicts = {}
for c in RUBRIC:
prompt = JUDGE_PROMPT.format(conversation=conversation, response=response, **c)
reply = call_model(prompt)
m = re.search(r"<result>\s*(PASS|FAIL|UNKNOWN)\s*</result>", reply, re.I)
verdicts[c["name"]] = m.group(1).upper() if m else "UNPARSEABLE"
return verdicts
The model call is left as a parameter so you can pass in your own client. A few details matter more than they look:
- Pass and fail are both written down. "Is the response helpful?" leaves the judge to invent the standard. "Fail if it asks for information already given" does not.
- UNKNOWN is a real answer. Count UNKNOWN verdicts separately. A criterion that often comes back UNKNOWN usually needs more context in the prompt, such as the tool results.
- Reasoning comes first, verdict last. The regex reads only the result tag, so the reasoning is there for you when a verdict looks wrong.
- Give a reference when correctness is checkable. If there is a right answer, put it in the prompt. That was the biggest single improvement in the MT-Bench math tests.
If you grade many transcripts against the same rubric, keep the fixed instructions at the top of the prompt and the transcript below them. That is the order prompt caching needs to reuse the shared part.
Comparing two outputs without position bias
When you compare two prompts or two models side by side, use the conservative approach from the MT-Bench paper: run the judge in both orders and count a win only when both runs agree. A disagreement becomes a tie.
from typing import Callable
def compare(judge: Callable[[str, str, str], str], question: str, a: str, b: str) -> str:
"""Ask a pairwise judge twice with the answers swapped. Returns 'a', 'b' or 'tie'.
judge(question, first, second) must return 'first', 'second' or 'tie'.
"""
one = judge(question, a, b) # a shown first
two = judge(question, b, a) # b shown first
a_wins = one == "first" and two == "second"
b_wins = one == "second" and two == "first"
return "a" if a_wins else "b" if b_wins else "tie"
A judge that always picks the first slot now produces only ties, which is the honest result. A judge that consistently prefers one answer still gets a clear winner. A high tie rate tells you the judge cannot separate the two, or that the rubric does not say what better means.
How do you know the judge agrees with people?
Label a sample by hand, run the judge on the same sample, and compare.
Husain recommends one principal domain expert whose judgment the product depends on, rather than a committee, and asks that person to write a short critique explaining each pass or fail. He says the critique should be detailed enough to reuse as a few-shot example in the judge prompt. On sample size, he suggests about 100 examples per failure mode and warns that below 60, the confidence intervals are often too wide to be useful.
Then measure agreement, and do not stop at percent agreement. If most outputs pass, a judge that passes everything looks accurate. Cohen's kappa corrects for that by comparing observed agreement with the agreement you would expect by chance from each rater's label rates.
from collections import Counter
def agreement(human: list[str], judge: list[str]) -> dict:
"""Compare a judge's pass/fail labels with a human's on the same items."""
assert len(human) == len(judge) and human
n = len(human)
p_o = sum(h == j for h, j in zip(human, judge)) / n # observed agreement
hc, jc = Counter(human), Counter(judge)
p_e = sum(hc[k] * jc[k] for k in hc) / (n * n) # agreement expected by chance
kappa = (p_o - p_e) / (1 - p_e) if p_e < 1 else 1.0
# Where they disagree matters as much as how often.
missed = sum(h == "fail" and j == "pass" for h, j in zip(human, judge))
false_alarm = sum(h == "pass" and j == "fail" for h, j in zip(human, judge))
return {"n": n, "agreement": round(p_o, 3), "kappa": round(kappa, 3),
"judge_passed_human_failed": missed, "judge_failed_human_passed": false_alarm}
# 20 transcripts a person labeled, and what the judge said about the same 20.
human = ["pass"] * 16 + ["fail"] * 4
lazy_judge = ["pass"] * 20 # passes everything
good_judge = ["pass"] * 15 + ["fail"] + ["fail"] * 3 + ["pass"]
print("lazy:", agreement(human, lazy_judge))
print("good:", agreement(human, good_judge))
Output:
lazy: {'n': 20, 'agreement': 0.8, 'kappa': 0.0, 'judge_passed_human_failed': 4, 'judge_failed_human_passed': 0}
good: {'n': 20, 'agreement': 0.9, 'kappa': 0.688, 'judge_passed_human_failed': 1, 'judge_failed_human_passed': 1}
The lazy judge agrees with the human 80 percent of the time and has a kappa of 0, because it learned nothing beyond "most things pass". The kappa values match scikit-learn's cohen_kappa_score on the same labels, so you can use that function instead once you have more than a toy example.
Look at the two disagreement counts as well. A judge that passes outputs the human failed is the dangerous kind, because those are the regressions your eval exists to catch.
Iterate on the disagreements
Each disagreement is either a rubric problem or a labeling problem. Read the judge's reasoning next to the expert's critique and decide which.
- If the judge applied the rubric correctly and still disagreed, the rubric does not say what the expert means. Rewrite the pass and fail lines.
- If the judge misread the transcript, add the expert's critique for that case as a few-shot example.
- If the expert changes their mind after reading the judge's reasoning, fix the label. The MT-Bench authors saw this too: when people were shown GPT-4's judgment, they were willing to change their own choice in 34 percent of cases.
Few-shot examples help, with a cost. In the MT-Bench study, adding them raised GPT-4's consistency under position swaps from 65.0 to 77.5 percent, made the prompt about four times as expensive, and, as the authors note, higher consistency does not guarantee higher accuracy.
Once the judge agrees with your expert at a level you are comfortable with, Anthropic's advice is that occasional human review is enough. Re-run the agreement check whenever you change the judge's model or prompt, since either one changes the grader you validated. The agent evals starter guide covers where the judge fits in a full eval suite, and the post on why agents call the wrong tool is a good source of criteria worth grading.
Checklist
- One criterion per judge call, with pass and fail both defined.
- PASS, FAIL or UNKNOWN, with reasoning before the verdict.
- A reference answer whenever correctness can be checked.
- A different model as judge from the one that produced the output.
- Pairwise comparisons run in both orders; disagreements become ties.
- Human labels on a sample, compared with kappa and both disagreement counts.
- Agreement re-checked after any change to the judge.
FAQ
What is LLM-as-a-judge?
It is using a language model to grade another model's output against written criteria, instead of a person or an exact-match check. It suits open-ended outputs such as answers, summaries and agent transcripts that a script cannot score.
Should an LLM judge use a 1 to 5 scale or pass/fail?
Pass/fail per criterion is easier to define, to act on and to check against human labels. Use a scale only when you need to rank outputs, and define what each score means.
How many human labels do I need to validate an LLM judge?
Hamel Husain suggests about 100 examples per failure mode, and says that below 60 the confidence intervals are often too wide to draw a useful conclusion.
How do I measure agreement between an LLM judge and humans?
Have a person and the judge label the same sample, then compute Cohen's kappa, which corrects for agreement by chance. Also count the cases where the judge passed something the person failed.
Can an LLM judge grade its own model's output?
It can, but the MT-Bench and G-Eval papers both saw signs that judges favor their own model's outputs. Anthropic's docs recommend grading with a different model from the one that produced the output.
Sources
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)
- Liu et al., G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023)
- Anthropic, Define success criteria and build evaluations (Claude API documentation)
- Anthropic, Demystifying evals for AI agents (engineering blog)
- Hamel Husain, Using LLM-as-a-Judge For Evaluation: A Complete Guide (2024)
- scikit-learn, cohen_kappa_score (API documentation)