Your A.I. Judge Has Favorites, Check For Them

You ask an AI to pick the better of two answers, and it picks the longer one every time — not the better one. Your judge has favorites, and if you don't know that going in, you'll trust scores that are measuring the wrong thing.

The problem: LLM-as-judge scales, but it isn't neutral

"LLM-as-judge" is the practice of using an AI model to grade another AI model's output — comparing two responses, scoring a single response against a rubric, or ranking a batch of outputs. It exists because manual human review doesn't scale: you can't have a person read every output from every prompt variant, every model version, every regression test. An LLM judge can run thousands of comparisons in the time a human reviewer does a handful.

But an LLM judge is still a model making a judgment call, and like a human grader, it has blind spots. The difference is that a human grader's biases are at least somewhat visible to a team that works with them daily. An LLM judge's biases are silent — the score just comes back as a number, and nothing about the number tells you it was influenced by something other than quality.

Why it works: three documented failure modes

• Positional bias. The judge tends to favor whichever answer it saw first (or, depending on the model, second), regardless of which one is actually better. Always present the same system's output as "Option A" and you're measuring position preference, not quality.
• Verbosity bias. The judge tends to favor the longer output, treating length itself as a signal of quality. A padded, repetitive answer can out-score a short, correct one purely because it says more.
• Self-enhancement bias. A judge model can favor an output because it recognizes stylistic patterns that match how it would have generated the answer itself — favoring "sounds like me" over "is actually better."

None of these mean the judge is useless. Bias doesn't mean the evaluation system is completely broken — it means you need to stay vigilant and test for it rather than trust it blindly.

How to run it: the positional-swap test

1. Run the comparison once, Answer A vs. Answer B, and record which one the judge picks.
2. Run the exact same comparison again, with the order swapped — B presented where A was, A where B was.
3. Compare the two verdicts. If the judge picks the same underlying answer both times, that's evidence the verdict reflects quality. If the winner flips, that's position talking, not quality.

The flow: A vs. B → Swap: B vs. A → Compare verdicts. This one test isolates positional bias cleanly, and it's cheap — one extra model call per comparison, automated as part of your eval harness.

To catch verbosity bias specifically, add a second variant: test a short, correct answer against a long, padded one where the padding adds no real information. If the judge favors length over correctness, either instruct the judge explicitly to ignore length, or normalize output lengths before comparison where the use case allows it.

Use cases

• Grading chatbot replies in an eval suite, before trusting the aggregate pass rate.
• Comparing two prompt variants (A/B testing prompts) where the "winner" determines what ships to production.
• Ranking model outputs at scale — a systematic position bias can silently favor whichever candidate happens to get slotted first in your pipeline.

Pitfalls

• Running the comparison once and trusting the number. A single-pass judge score, with no swap test, tells you almost nothing about whether position or length drove the result.
• Assuming the swap test is a one-time validation. Bias can shift when you change judge models, prompts, or rubrics — re-run the check whenever any of those change.
• Conflating "biased" with "broken." A biased judge, once you know its bias, can still be useful — you can correct for a known positional or verbosity skew. An unexamined judge, you can't correct for anything.
• Only testing positional bias and skipping verbosity/self-enhancement. They're independent failure modes.

Power tricks

• Automate the swap as part of the eval harness rather than a manual spot-check — run both orders every time, always.
• Pair the swap test with a short-vs-padded pair to isolate verbosity bias specifically, not just positional bias.
• Track disagreement rate across swaps as a metric of its own — a rising disagreement rate over time is an early signal your judge setup needs recalibration.

Resources

• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023)
• OpenAI Evals — model-graded evaluation reference