LLM-as-a-Judge Pitfalls: Flaky Scores, Self-Preference, Grading on a Curve
An LLM judge can pass 80%+ agreement with human raters and still lie to you in specific, documented ways: non-deterministic scores, self-preference bias, leniency, and overfitting to the judge itself. Here is what independently-verified research says about each pitfall, and the
