Field guide · AI Agents
How to Evaluate AI Agents: Building a Test Set You Can Trust
Eleven lessons on detecting change rather than measuring intelligence: real cases pulled from traffic, scoring picked per case, and the failures a pass rate never reports.
- Lessons
- 11
- Read
- 22 min
- Figures
- 8
AI-drafted, reviewed by Muhammad Qasim Hammad on November 13, 2026. See our AI disclosure.
Table of contents
- What you will end up with
- 1. Decide what working means before you measure anything
- 2. Collect the first twenty cases from what already happened
- 3. Write the expected outcome, not the expected words
- 4. Pick a scoring method per case, not per project
- 5. Use a model as a judge, carefully
- 6. Score the steps, not just the final answer
- 7. Run it as a regression, not a one-off
- 8. Watch the things a pass rate never catches
- 9. Close the loop with production
- 10. Know when the set is lying to you
- 11. Decide what a failing eval actually blocks
- What I would do differently
- Where to go next
Most teams find out their agent got worse from a customer, not from a test. The model version moved, a prompt was tightened, a tool description was reworded, and nothing anywhere said the quality dropped. This guide builds the thing that would have said it: a small set of real cases, scored the same way every time, run before every change.
The obstacle is rarely effort. It is that evaluation looks like it needs a framework, a labelled dataset and a metric nobody can define, so it gets postponed until there is a problem, which is exactly when there is no time to build it.
What you will end up with#
A set of twenty to fifty cases drawn from real traffic, a scoring method chosen per case rather than per project, a run you can trigger before merging a change, and a short list of the failures a score will never catch. Eleven lessons.
What this does not cover: academic benchmarks, leaderboard chasing, or training your own model. This is the evaluation a small team can actually keep running after the week it was built.
One idea carries the whole guide: you are not measuring intelligence, you are detecting change. That is a far easier problem, and it is the one that actually protects you.
| Lessons | What you get | Roughly |
|---|---|---|
| 1 to 3 | A definition of working, and the first real cases | 20 min |
| 4 to 6 | Scoring that survives contact with fuzzy output | 25 min |
| 7 to 9 | Running it as a gate, and the parts a score misses | 20 min |
| 10 to 11 | When the set is lying, and what a failure blocks | 15 min |
Lesson 1Decide what working means before you measure anything#
Ask what the agent is for, in one sentence, with an outcome in it. "Answers billing questions" is not an outcome. "Resolves a billing question without a human touching it, and does not invent a refund policy" is, because both halves can fail independently and you would want to know which one did.
Write that sentence down. Everything else in this guide is downstream of it: the cases you collect, the way you score them, and the failures you agree to care about. Teams that skip it end up measuring what is easy to measure, which is usually output format.
Split the sentence into checks that fail separately:
- Did it finish? The agent produced an answer or completed the action, rather than looping, timing out, or refusing.
- Was it right? The content matches what a competent person would have said or done.
- Did it stay inside the rules? No invented policy, no tool it should not have called, no data it should not have shown.
Three checks, not one score. A single number that blends them tells you the agent is worse without telling you which part broke, and the fix for each is different. Formatting problems are prompt work, wrong answers are usually retrieval or tool work, and rule breaks are guardrail work.
The table is worth arguing with before you accept it. If your agent's real job is triage rather than answering, "was it right" means the category was right and nothing else matters. Get this wrong and you will spend weeks improving prose in a system whose only job is to route.
Lesson 2Collect the first twenty cases from what already happened#
Do not write test cases from imagination. You will write the ones the agent already handles, because those are the ones you can picture. Pull real inputs instead: the last two weeks of requests, the tickets that got escalated, the conversations that ran long, and anything a colleague forwarded with the word "look".
Aim for twenty to start. Twenty real cases will find more than two hundred invented ones, and you can read twenty in a sitting, which matters because you have to agree on the expected outcome for each.
Weight the set deliberately rather than sampling evenly:
- Ten ordinary cases. The traffic you actually get. These catch regressions in the common path, which is where damage is widest.
- Six known-hard cases. Ambiguous requests, missing information, two questions in one message.
- Three rule tests. Inputs that tempt the agent past a boundary, including at least one attempt at prompt injection.
- One case you know it fails. A permanent reminder that the suite is honest.
That last one gets removed more often than any other, usually with a note that it is not representative. Keep it. A suite with no failing case has stopped being a measurement and become a ritual.
If you have no production traffic yet, generate the set instead, but treat the generated cases as provisional and replace them as real ones arrive. The trap with generated inputs is that they carry the model's idea of what users ask, which is tidier and better spelled than the real thing, and synthetic eval sets drift away from live traffic in exactly that direction.
Lesson 3Write the expected outcome, not the expected words#
For each case, record what a correct response contains, not the sentence you hope to see. Exact text is unusable as an expectation: two answers can differ in every word and both be right, and a model that matches your phrasing has probably just been prompted into it.
Store each case as a small record with the input, the expectation, the reason it is in the set, and its type. That last field is what lets you say "the hard cases are fine, the ordinary ones broke", which is a much more useful sentence than a pass rate.
Expectations come in a few shapes, and picking the shape is most of the work:
| Expectation shape | Use it when | Example |
|---|---|---|
| Exact value | The output is an ID, a category, a number | Ticket routed to billing |
| Required facts | The answer must contain specific things | Names the 30-day window and the exception |
| Forbidden content | A rule must hold | Never states a refund amount |
| Tool call | The action matters more than the prose | Calls lookup_order exactly once |
| Human judgement | Genuinely subjective | Tone is appropriate for a complaint |
A case record can be a row in a spreadsheet or a small file. The shape matters more than the format, and this is the minimum that stays useful a year later:
id: billing-refund-window-01
type: ordinary
input: "I bought this 3 weeks ago, can I still send it back?"
expect:
contains: ["30 day", "return"]
forbids: ["refund of $", "no returns"]
tools: ["lookup_order"]
why: "Most common billing question. Agent invented a policy here in June."The why field is the one people leave out and the one that saves the suite. Two months on, a case fails and nobody remembers whether the expectation was deliberate or an accident, so it gets loosened. A sentence of provenance prevents that.
Most cases want two of these at once: required facts plus forbidden content is the workhorse pair for anything customer-facing. Only reach for the last row when the first four genuinely cannot express what you mean, because it is the row that costs the most to score.
A useful discipline here is writing the expectation before you look at what the agent currently does. Write it after, and you will unconsciously describe the current behaviour, which makes the case incapable of ever failing.
Lesson 4Pick a scoring method per case, not per project#
Most eval projects stall on choosing one scoring method for everything, which is impossible because the cases are not alike. Routing to a category is a string comparison. Whether an answer covered the refund window is a keyword or model check. Whether the tone suited an angry customer is a person reading it.
Score each case with the cheapest method that can actually fail correctly. A method that never fails is worthless, and a method that fails at random is worse than nothing, because it trains you to ignore the suite.
The four methods, cheapest first:
- Exact or normalised match. Compare after lowercasing and trimming. For categories, IDs, booleans and numbers. Runs in milliseconds and costs nothing.
- Contains and does-not-contain. A list of required strings and a list of forbidden ones. Crude, but it catches the two failures that actually hurt: a missing fact, and a policy the agent made up.
- Model as judge. Another model reads the answer against a rubric and returns a verdict. Use it where correctness is real but unstateable as a string.
- A person. Slow, expensive, and the only honest option for tone, tact, and whether an answer would embarrass you.
Aim for most cases on rows one and two. A suite that is 80 percent string checks runs in seconds for free, which is what makes it something you actually run before every change rather than once a quarter.
Lesson 5Use a model as a judge, carefully#
A judge model is useful and it is also the part of the suite most likely to quietly lie to you. It agrees with itself less than you expect, it prefers answers written in its own style, and it drifts toward the middle of whatever scale you give it.
Three habits make a judge trustworthy enough to gate on:
- Ask for a verdict, not a rating. "Does this answer state the 30-day window? yes or no" is reproducible. "Rate this answer 1 to 10" is not, and the scores will cluster around seven regardless of quality.
- Give it the expectation, not just the answer. A judge scoring in a vacuum invents its own standard. A judge handed your required-facts list is doing comparison, which models are far better at.
- Check the judge against yourself once. Read thirty judged cases by hand and count the disagreements. If it is more than a handful, the rubric is vague, not the model.
Never let one model judge its own family's output when you are comparing providers. Self-preference is real and it will make your incumbent look better than it is. The broader set of traps is worth reading before you rely on a judge: LLM-as-a-judge fails in predictable ways, most of them invisible in the score itself.
There is also a case for skipping the judge entirely. If you have no labels and no rubric you trust, what you can still measure without ground truth is narrower than a judge implies but far more reliable: consistency between runs, refusal rate, latency, and whether the agent contradicts itself.
Lesson 6Score the steps, not just the final answer#
An agent that reaches the right answer through four wrong tool calls is not working. It is expensive, slow, and one prompt change away from reaching the wrong answer. Final-answer scoring hides all of that, and it is the single biggest gap in most eval suites.
Capture the trace for every case: which tools were called, in what order, with what arguments, and what came back. You need this to debug anyway, and once you have it, three more checks become cheap.
| Step check | Catches | Cost to add |
|---|---|---|
| Tool call count | Loops, retries, runaway cost | Trivial |
| Which tools were used | Wrong tool for the job | Trivial |
| Arguments passed | Malformed or hallucinated parameters | Small |
| Retrieved documents | Retrieval that missed, answer that guessed | Small |
In practice the step checks read like assertions against the stored trace, and they are worth writing even when they look trivially obvious:
assert trace.tool_calls("lookup_order") == 1 # not zero, not four
assert "delete_order" not in trace.tools_used # never, under any input
assert trace.retrieved_ids contains "policy-returns-v3"
assert trace.total_ms < 8000The second line is the one that earns its keep. It never fails until the day someone adds a destructive tool to the shared tool set and every agent inherits it, and on that day it fails on every case at once, which is exactly the alarm you want.
Tool selection deserves its own attention because the fix is usually not the agent at all. When an agent reaches for the wrong tool, the cause is nearly always the description it was given rather than its reasoning, and tool descriptions decide tool-use reliability more than model choice does.
If the agent retrieves before it answers, score the retrieval separately. An answer can be wrong because the model reasoned badly or because it was handed the wrong three documents, and those have nothing in common as problems. The four metrics in a usable retrieval scorecard separate the two cleanly, and running them costs nothing extra once the trace is stored.
The practical shape of all this is that your case record grows a second half. The first half is what came out. The second is what happened on the way, and that is where the early warnings live: a tool count creeping up, a retrieval hit rate slipping, an agent starting to ask for clarification more often than it did last month.
Lesson 7Run it as a regression, not a one-off#
An eval you run when you are worried tells you nothing, because you only run it after something already went wrong. The value comes from running it on every change, so that the first time a number moves you know exactly which change moved it.
Wire it to the four moments that actually change agent behaviour:
- Prompt edits. The most common cause of silent regressions, and the easiest change to make without thinking of it as a change.
- Model version changes. Including the ones you did not choose, when a provider updates a model behind a stable name.
- Tool changes. A renamed parameter, a new tool added to the set, a description reworded for clarity.
- Knowledge changes. New documents indexed, old ones removed, a chunking setting adjusted.
Store every run: the date, the commit or prompt version, the per-case results, and the aggregate. The comparison between runs is the product here. A pass rate of 84 percent means nothing on its own and everything if last week it was 96 percent.
Version the prompt alongside the code so a run can name what it tested. A result that says "84 percent" is an anecdote. A result that says "84 percent, prompt v14, model pinned to a dated version, 22 September" is evidence, and it is the difference between finding a regression in an afternoon and arguing about whether one happened. Treating prompts as versioned artefacts rather than configuration is the whole point of prompt versioning.
Keep the suite fast enough to be boring. If a run takes twenty minutes, people will skip it before a small change, and small changes are where regressions hide. Cache what you can, run cases in parallel where the API allows, and accept a rough rate limit ceiling as the real constraint on suite size.
One warning about promotion. When a run fails, the temptation is to fix the case rather than the agent, especially under deadline. Adjusting an expectation because the new behaviour is "also fine" is sometimes correct and is usually how a suite dies. Make the edit visible: change it in a separate commit from the code change, with a sentence saying why.
Lesson 8Watch the things a pass rate never catches#
A suite can be green while the agent is getting worse in ways nobody wrote a case for. Cost per run creeping up, latency doubling, the refusal rate climbing, an answer style that technically passes but reads badly. None of these fail a check, and all of them reach users.
Track four numbers alongside the pass rate, from the same runs, at no extra cost:
| Signal | What a rise usually means | Where it shows first |
|---|---|---|
| Tokens per case | Context bloat, or a retry loop | The bill |
| Wall-clock per case | Extra tool hops, a slower model | User complaints |
| Refusal or fallback rate | Over-tightened guardrails | Support volume |
| Answer length variance | Prompt drift, unstable formatting | Downstream parsing |
Cost in particular deserves a hard number rather than a trend line, because the sensible unit is not tokens. Work out what the agent costs per resolved request and compare it to what the request cost before, which is the framing in cost per resolved ticket and the only version of the number a business will act on.
The same runs give you a cheap read on stability. Run the same case three times and see whether the verdict changes. An agent that passes two runs in three is not passing, and averaging it into a percentage is how that gets hidden. If a case is genuinely nondeterministic, either the expectation is too tight or the temperature setting is doing something you did not intend.
Lesson 9Close the loop with production#
The suite starts as a snapshot of two weeks of traffic and immediately begins to age. Users find new phrasings, your product ships new features, and the cases you wrote in month one stop resembling the requests arriving in month four.
Feed it from live traffic on a schedule you actually keep. Once a month, take twenty minutes and do three things:
- Add the escalations. Every request that a human had to take over is a case the agent failed. Add the five most representative, with the outcome the human produced as the expectation.
- Add the complaints. Anything a user pushed back on, even when the agent was technically correct. These become the tone and completeness cases you would never have invented.
- Retire the dead ones. Cases about a feature you removed, or a phrasing nobody uses. A stale case that always passes is noise in the denominator.
You need production traces for step one to be possible at all, which is the practical argument for tracing every agent run even when nothing is wrong. Without it you have complaints without inputs, and a complaint you cannot reproduce cannot become a case.
Watch the shape of the set as it grows. A suite that is 80 percent edge cases has stopped measuring the product, because the ordinary path now contributes almost nothing to the score and can break without moving the number. When hard cases pass a third of the time and ordinary ones pass always, split the two into separate reported numbers rather than one blended average that describes neither.
One more source is worth mining: the requests the agent handled without complaint but slowly, or expensively, or after three tool calls that should have been one. Nobody escalates those, so they never become cases, and they are where quality decays without anybody filing anything.
There is a version of this that runs continuously instead of monthly: sample a small percentage of live requests, score them with the same rubric, and alert on the trend. It is more work to build and it is the right end state, because it measures the thing users experienced rather than the thing you remembered to test.
Lesson 10Know when the set is lying to you#
Every eval suite eventually starts reporting a number that no longer tracks reality. It happens quietly, and the tell is always the same: the score stops moving while your sense of the agent keeps changing. Learn the four ways it happens, because each has a different repair.
The failure that costs the most is overfitting to the suite. After a few months of tuning prompts against the same fifty cases, the agent gets very good at those fifty and no better at anything else. The score rises, users notice nothing, and the suite has quietly become the target rather than the measurement.
Two habits keep it honest:
- Hold back a set you never tune against. Ten cases, sealed, run only when you want a real reading. If the main suite improves and the held-back set does not, you have been tuning to the test.
- Rotate cases in and out. Replacing a fifth of the set each quarter costs an hour and resets the drift.
The other three are simpler. A suite with no failures has expectations too loose to fail. A suite where everything fails has expectations written against an aspiration rather than the agent. And a suite nobody has looked at in two months is measuring a version of the product that no longer exists.
Lesson 11Decide what a failing eval actually blocks#
An eval that reports is a dashboard. An eval that blocks is a gate. Most teams need the second and build the first, because deciding what a red run prevents is a policy question rather than a technical one, and it is easier to leave undecided.
Write the policy down as three tiers, because "the suite failed" is not one situation:
- Rule failures block, always. A case that tests a boundary. Invented policy, wrong data shown, a tool called that should not have been. One of these fails and the change does not ship, regardless of the aggregate.
- Ordinary-case drops block below a line. Pick the line before you need it. A drop of more than a few points on the common path means the change gets investigated, not shipped.
- Hard-case movement is information. These fluctuate. Record it, look at it monthly, and do not gate on it, or the gate will be ignored within a fortnight.
The tiering matters more than the thresholds. A single blended pass rate forces you to pick one number that is simultaneously too strict for the hard cases and far too loose for the rule tests, and whichever way you set it, the gate ends up either blocking everything or catching nothing.
In practice the policy fits in four lines next to the suite, and writing it there rather than in someone's head is most of the value:
rule cases -> any failure blocks the change
ordinary cases -> more than 5 points below last run blocks
hard cases -> recorded, reviewed monthly, never blocks
held-back set -> run before a release, not on every changePick the five-point line by looking at your own history rather than borrowing it. Run the suite three times against an unchanged agent and see how much it moves on its own. That spread is your noise floor, and a threshold set below it will fire constantly on nothing, which is the fastest way to teach a team to ignore a gate.
Attach an owner and a response, not just a threshold. A red run with nobody responsible becomes a warning people click past, and that is worse than no gate at all, because it teaches the team that failures are cosmetic. The same reasoning applies to how much failure you tolerate in production, which is the argument for setting an explicit error budget rather than an implicit one.
What I would do differently#
I would build the twenty cases before building the agent, not after. Written first, the expectations describe what you wanted; written afterwards, they describe what you got, and the difference is invisible from inside the file.
I would also stop trying to produce one number. The blended pass rate was the thing everyone asked for and the thing that hid every problem worth knowing about, and splitting it into ordinary, hard and rule tiers took ten minutes and made every subsequent conversation shorter.
Where to go next#
If you are building the agent alongside the evaluation, the fuller build sequence is in how to build an AI agent, and the evaluation belongs at step one of it rather than at the end.
If your agent runs on a workflow platform, the mechanics of storing cases, mocking tools and running the set on a schedule are covered concretely in testing and evaluating n8n AI agents. If the agent's failure mode is confident invention rather than a wrong tool, start with why agents hallucinate and what actually reduces it and add citation checks to your required-facts expectations, following source grounding.
And when a new model release tempts you to swap, the suite you just built is the thing that makes the decision boring: run it, compare, decide. That is the whole argument in how to evaluate a new model release.
Frequently asked questions
How many test cases does an AI agent evaluation need?
Can you evaluate an AI agent without labelled data?
Should a failing evaluation block a deployment?
Why does an agent's evaluation score stop reflecting reality?
Sources
Primary references and vendor documentation used while drafting and reviewing this article.
Written by
Muhammad Qasim Hammad is an AI agent and automation expert and the founder of Cart Gaze LLC (cartgaze.com). He builds product for the love of it: when an idea lands, a working prototype is usually running within hours, built with the same AI agents and automations he sells. He puts his own output at roughly 20× what it was before agents, and the Agentic OS behind this site is the working proof, documented in public with the tools he actually ran and what they really cost.
AI & Automation Services
Want a pipeline like this running in your business?
I'm Qasim — I design and ship AI agents and n8n automations for solo operators and small teams. Tell me what's eating your team's week, and I'll scope a fix.
Related reading
How to Test and Evaluate n8n AI Agents: Accuracy, Mocks and Regression
An AI agent is non-deterministic, so 'it worked when I tried it' is not testing. Use n8n's built-in Evaluation feature, a golden test set, and the right scoring metrics to test n8n AI agents and prove a change made it better, not worse.
Synthetic Data for AI Agent Evals: Build the Set Before You Have Users
Real production traces are the best eval data you can get, and most teams do not have enough of them yet. Here is the concrete mechanism for generating a synthetic eval set from a handful of real examples, plus the honest failure modes: data that drifts from what real users do,
How to Build an AI Agent: The 80% That Survives Week Two
Most tutorials get you a working agent in ten minutes and skip what breaks it in week two. Sixteen lessons covering the loop, tool descriptions, memory, fallbacks, guardrails, evaluation, and the cost levers that decide your bill.
AI Agent Tool Design: What to Expose and How to Shape It
When an agent picks the wrong tool or loops on a lookup, the cause is rarely the model. Eleven lessons on tool boundaries, descriptions that resolve ambiguity, schemas that make bad calls hard to express, and why removing a tool improves accuracy.
RAG Evaluation: How to Measure If Your Retrieval Is Actually Good
Your RAG pipeline feels okay, but no number says if retrieval is good. This is the reproducible scorecard: context precision and recall for retrieval, faithfulness and answer relevancy for generation, scored 0 to 1 with Ragas, plus which metric to read when an answer is wrong.
LLM-as-a-Judge Pitfalls: Flaky Scores, Self-Preference, Grading on a Curve
An LLM judge can pass 80%+ agreement with human raters and still lie to you in specific, documented ways: non-deterministic scores, self-preference bias, leniency, and overfitting to the judge itself. Here is what independently-verified research says about each pitfall, and the





