#ai agent evals

Real production traces are the best eval data you can get, and most teams do not have enough of them yet. Here is the concrete mechanism for generating a synthetic eval set from a handful of real examples, plus the honest failure modes: data that drifts from what real users do,

Muhammad Qasim Hammad·August 29, 2026·11 min

An LLM judge can pass 80%+ agreement with human raters and still lie to you in specific, documented ways: non-deterministic scores, self-preference bias, leniency, and overfitting to the judge itself. Here is what independently-verified research says about each pitfall, and the

Muhammad Qasim Hammad·August 22, 2026·11 min