LLM-as-a-Judge Pitfalls: Flaky Scores, Self-Preference, Grading on a Curve

You wire an LLM judge into your eval pipeline, watch the pass rate climb, and ship with confidence. Then a real user hits a case where the answer is wrong in a way a careful person would catch in seconds, and your judge scored it a 9 out of 10 the whole time. LLM-as-judge pitfalls are the specific, documented ways an LLM grader drifts from the truth, and most of them are invisible until you go looking for them.

What is LLM-as-a-judge, and why does it fail in predictable ways?#

LLM-as-a-judge means using a language model to grade another model's output against a rubric or a reference answer, instead of paying a human reviewer for every case. The 2023 MT-Bench study found GPT-4 judges agree with human preferences over 80% of the time, but that number hides several repeatable failure patterns underneath it.

That agreement rate comes from Zheng et al.'s paper introducing MT-Bench and Chatbot Arena as benchmarks for LLM judges, and it named three biases that still show up in every later paper on the topic: position bias, where the order you present two answers changes the verdict; verbosity bias, where a judge favors the longer answer with quality held constant; and self-enhancement bias, where a judge favors output that resembles its own style. Source

If you already run evals in n8n, you have probably met one of these without naming it. Our guide to testing and evaluating AI agents in n8n covers the built-in Correctness and Helpfulness metrics, and both are LLM-as-judge under a different name, which means both inherit every pitfall below.

Every number here is read directly from the cited paper, not a summary of a summary.

Six pitfalls turn up across the research often enough to be worth a standing reference. Use the table to jump straight to the one that is biting you right now.

Each row below gets its own section, roughly in the order it tends to surprise a small team running its first LLM-judge pipeline.

Why do two runs of the same judge disagree with each other?#

Non-determinism means the same input, the same model, and the same temperature setting can still produce a different score on a second run. A February 2026 study tested five judge models on real retrieval-augmented generation question-answer pairs and found meaningful score variability even at temperature 0, which is supposed to be the deterministic setting.

The study covered GPT-4o, GPT-4o-mini, Gemini 2.5 Flash, Claude Haiku 4.5, and Claude Sonnet 4.5, and found completeness scoring drifted the most of the metrics tested. Lowering the temperature stabilized GPT-4o and Gemini, but had inconsistent effects on the Claude models, so the fix is not as simple as setting temperature to 0 and moving on. This is a single, recent study rather than an established consensus, so treat the exact numbers as directional and worth re-checking as more work replicates it. Source

If you have not looked at how temperature and top-p actually shape an LLM's output, our temperature and top-p explainer covers the sampling mechanics this finding depends on. Treat non-determinism as a floor, not a rounding error: if your eval suite reports one score per case, you cannot tell a real regression from noise in the judge itself. Run every case at least twice before you trust a drop in the number.

Why does a judge favor its own model family's answers?#

Self-preference bias is a judge scoring an answer higher because it resembles text the judge model itself would generate, not because the answer is actually better. A 2024 NeurIPS workshop paper found GPT-4 shows a significant degree of this bias, and traced the mechanism to perplexity: judges rate familiar-sounding text higher, regardless of who wrote it.

Zheng et al. named this self-enhancement bias back in 2023; the 2024 paper gave it a sharper name and, more usefully, a way to measure it. That familiarity effect matters the moment you swap models for a comparison. Evaluate a Sonnet-generated answer with a Sonnet judge, then evaluate a GPT-generated answer with that same Sonnet judge, and the comparison is no longer fair by construction, because one output is more familiar to the grader than the other. Source

The practical fix is boring but it works: never let the model you are testing also be the model doing the grading, especially when the whole point of the eval is choosing between vendors. A neutral third model as judge costs you one extra API integration and removes an entire category of skewed comparison.

Why do judges keep grading on a curve?#

Leniency bias is a judge's tendency to score a borderline answer as correct rather than fail it, even when its own stated criteria are not fully met. A study testing 13 judge models found this leniency probability ranged from 0.36 to 0.94 depending on the model, meaning some judges pass questionable answers most of the time by default.

Leniency bias means a judge defaults toward calling an answer correct instead of failing a borderline one.

That same research tested how much rubric detail actually helps. Four grading-prompt variants, from a bare instruction to a full rubric with worked examples, showed only the strongest judge models got more aligned with more detail; weaker judges actually drifted further from human scores as the instructions got longer. That is a narrower and more useful finding than "too much context confuses a judge": it is specifically about how much grading instruction you hand the judge, not how long the thing being graded is. Source

The takeaway is not "write a longer rubric and move on." It is: test your rubric against a held-out human-graded set before you assume more instructions helped, because for a weaker or cheaper judge model, they might have quietly made scores less accurate instead of more.

What is overfitting to the judge, and why is it Goodhart's Law in disguise?#

Overfitting to the judge happens when you repeatedly tune a prompt or a model against your own LLM grader until the judge score climbs, while real quality for actual users stays flat. It is Goodhart's Law applied to evals: once a score becomes the optimization target, it stops measuring what you actually care about.

The channel is usually one of the biases above. A prompt tuned against judge feedback learns to produce longer answers because of verbosity bias, or answers that read like the judge's own house style because of self-preference bias, since those are the properties the judge happens to reward, not because the answers serve users any better. Every metric that becomes a target eventually gets gamed, on purpose or by accident, and an LLM judge is a metric like any other.

Guard against it the way you would guard against any proxy metric: keep a small, untouched human-graded set that you never train or prompt-tune against, and check it on a schedule. If the judge score keeps climbing while that held-out set stays flat or drops, you are optimizing for the grader, not the product it is supposed to stand in for.

How do you actually catch these pitfalls before they wreck a pipeline?#

You catch these pitfalls with a handful of concrete tests, not by reading about them once and trusting your setup forever. Swap answer order, strip both answers to equal length, reword the prompt neutrally, and re-run the identical case twice: if the score moves on either of these, you have found a live bias, not a stable measurement.

Six cheap tests that surface a live bias instead of a stable measurement.

Run the tests in roughly this order, since each is cheap and each isolates a different pitfall: reorder the two answers being compared and check for a flipped verdict, pad or trim both answers to matching length and check the score holds, swap in a different judge model and see if the ranking changes, and grade 20 to 30 cases yourself so you have a human baseline to calibrate against. None of this needs to be elaborate. A spreadsheet and an afternoon beats a judge score nobody has ever questioned.

Should you trust a single judge's score?#

A single judge is fine for quick iteration, where a wrong verdict costs you a re-run, not a bad decision. For anything higher stakes, a release gate, a vendor comparison, a customer-facing quality claim, pair the judge with a held-out human-graded sample, and consider a panel of several smaller judges instead of one large one.

A panel of smaller, diverse judges matched or beat a single large judge at a fraction of the cost, per the PoLL study.

One study tested a panel of smaller, diverse judge models against a single large judge like GPT-4, across three judge settings and six datasets. The panel matched or beat the single judge on human correlation, showed less bias because its models came from different families, and ran at over 7x lower cost. Source

Three checks decide whether the score is worth trusting: calibration, stability under reordering, and single-judge risk.

Calibration is the step teams skip because it feels like extra work before the real work starts. Grade a sample against human labels before you trust any judge, single or panel, and re-check that calibration whenever you swap the judge model or the prompt it grades against. If your test cases are synthetic rather than pulled from real usage, our guide to synthetic data for AI agent evals covers how to keep that data honest enough to calibrate against in the first place.

What should you set up this week?#

This week, take your existing LLM-as-judge setup and run it through the four cheap tests: reorder answers, trim to equal length, swap the judge model, and re-run one case twice. If either test moves the score, you have found your first real bug, and fixing that is worth more than adding a fifth pitfall to worry about.

Then decide whether you need a single judge or a panel, based on what is actually at stake. A judge score that only gates your own iteration speed can stay simple. A judge score that decides what ships to a customer, or which vendor you pay for, earns the calibration pass and the panel.

If evals are new to your stack, start with testing and evaluating AI agents in n8n for the harness itself. Once a bad run makes it to production anyway, AI agent observability is the other half of the story: the trace that shows you what the judge never had a chance to catch.