How to Test and Evaluate n8n AI Agents: Accuracy, Mocks and Regression

You tweak the system prompt, the demo looks great, and you ship it. A week later three other cases are quietly broken, and you only find out when a user complains. An AI agent is non-deterministic, so the same input can produce a different answer run to run. To test n8n AI agents reliably you need a repeatable scored run over a fixed set of labelled cases, not one-off manual pokes at the chat window.

"It worked when I tried it" proves one path worked once. It says nothing about the other twenty paths your last edit might have moved.

Why does an AI agent need an eval harness, not just a manual test?#

A manual test checks one input on one run of a system that can answer differently next time. An eval harness runs many labelled cases every time you change something, so a prompt edit that fixes one case and breaks three others shows up as a lower score instead of a silent regression you discover in production a week later.

The shift is the same one that made unit tests standard. You stop trusting that a change is safe because the demo looked right, and you start proving it against a fixed set of cases. Anthropic frames evals as empirical and continuous, the continuous integration layer for an agent, and the logic is identical here: no green run, no change to production.

You do not need a research benchmark to start. Ten to twenty hand-labelled cases drawn from real usage, including the ones that have bitten you before, already beat zero. A small honest test set is the cheapest upgrade you can make to an agent you actually depend on.

How does n8n's Evaluation feature work?#

n8n ships a built-in Evaluation feature: an Evaluation trigger that reads your test dataset one row at a time, plus an Evaluation node that records results and scores. You point the trigger at a Data table or a Google Sheet of cases and run each row through your existing agent, no separate tooling required.

The trigger reads a Data table (with an optional row limit and filter) or a Google Sheet through a credential, and emits one item per row. Each row holds the input and usually the expected output; you leave the actual-output column blank for the run to fill. The Evaluation node then gives you three operations, named verbatim in the docs:

• Set Outputs writes the agent's actual result back to the dataset so it can be compared to the expected answer.
• Set Metrics calculates and records named scores to the Evaluations tab, where they chart over time.
• Check if Evaluating branches the workflow so production traffic and test traffic take different paths.

The harness itself is just a normal workflow: Evaluation trigger, then your AI Agent (chat model plus tools), then Set Outputs, then Set Metrics. Run it from the canvas or the Evaluations tab and you get saved, charted results you can line up version against version.

The harness is just a normal workflow the Evaluation trigger replays one row at a time.

How do you build a golden test set?#

A golden test set is a curated table of real inputs paired with confirmed-correct outputs. Build it from production: sample real executions from n8n's history, review them, label the right answer, and keep that labelled subset as your benchmark. Then grow it by adding every new failure you find as a permanent case.

Two rules keep the set honest. First, include the hard cases on purpose: edge cases, previous failure points, and adversarial inputs like sarcasm, mixed signals, or a competitor's name. A set of only easy cases hides the bugs you most need to catch. Second, let it compound. Every time the agent fails in production, add that exact input with its correct answer, so a failure caught once can never silently return.

Keep an input column and an expected-output column, and leave the actual-output column blank. The run fills it through Set Outputs, and the metric compares the two. Production-derived cases beat synthetic ones because they match the distribution your agent actually sees.

Pick the metric by the shape of the output, not by habit.

Which scoring method should you use?#

Match the metric to the shape of the output. Fixed labels score with Categorization (exact match). Short strings that should nearly match a reference score with String similarity. Open-ended text needs an LLM-as-judge metric. Tool decisions need the Tools Used metric plus a trajectory check. Most real agents combine a deterministic metric with an LLM judge.

n8n ships five built-in metrics, and the right choice follows the output type.

Use deterministic metrics where the output shape allows; reserve the LLM judge for open-ended text.

An LLM-as-judge metric scores an output the way a careful reviewer would, fast and at scale, which is why it fits open-ended answers. Pair it with a clear rubric, the way Anthropic suggests: "the answer must mention X in the first sentence or it is graded incorrect." But a judge is itself a model call. It costs tokens and can be miscalibrated, so calibrate it against a handful of human-graded cases before you trust it, and prefer a deterministic metric whenever the output shape allows. When the built-ins are not enough, you can return a custom score (JSON validity, latency, your own judge) from a sub-workflow and record it with Set Metrics.

How do you mock tools and run regression safely?#

Use the Evaluation node's Check if Evaluating operation to split the workflow into a production path and an evaluation path. Route side-effecting nodes (email, CRM writes, payments, Slack) down the production branch only, so a 50-case test run does not fire 50 real emails. Without that branch, the harness is too dangerous to run often, which defeats the point.

Without this branch, a 50-case eval run sends 50 real emails. The branch is the mock.

For a tool-using agent, mock the tool's result too, not just the side effect. Feed the agent a deterministic observation so you are testing its reasoning rather than a flaky upstream API. Then run regressions with one discipline: change exactly one variable at a time.

The loop is simple. Take a baseline run on your golden set, change one thing (the prompt, the model, or a tool), re-run the same dataset, and compare scores side by side in the Evaluations tab. Swap the model and the prompt together and you will not know which one moved the score. Track more than accuracy, too: n8n automatically records token count and execution time, so pair those deterministic measurements with a quality metric to catch a model that scores the same but runs slower or costs more.

Change one variable at a time, or you will not know which change moved the score.

Run the suite on a cadence, daily for high-volume agents and weekly for quieter ones, so drift surfaces before users report it. And treat the run as the gate: no green run, no prompt change to production.

What does evaluation cost? (modeled)#

Deterministic metrics (Categorization, String similarity, Tools Used) cost nothing in tokens. The only token cost is an LLM-as-judge metric, because each graded case is an extra model call. The figures below are modeled from June-2026 Claude list prices and a stated per-case assumption, so you can redo the arithmetic by hand; they are not a measured bill.

Assume each judge call is about 1,500 input tokens (the case, the reference answer, and the rubric) plus 150 output tokens (a short verdict). Grading 100 open-ended cases is then roughly 150,000 input and 15,000 output tokens. Check current Claude pricing before relying on these, since rates change.

Self-hosted n8n adds $0 in platform cost, and the agent's own run cost applies either way. Two levers cut the judge bill further on large suites: run the judge on Haiku, and cache the rubric and reference so repeated reads bill at a fraction of full input. The prompt caching guide covers that, and Claude vs GPT vs Gemini in n8n agents helps pick the judge model.

What should you set up this week?#

This week, build the smallest honest harness you can. Pull ten to twenty real inputs from your agent's execution history, label the correct answer for each, and drop them in a Data table. Add an Evaluation trigger and one metric that matches your output, and run the baseline once.

Then make it your gate. The next time you touch the prompt or swap the model, re-run the same set, compare the scores, and only ship if they hold or improve. Add every new production failure to the set as you go, and the harness gets stronger every week instead of going stale.