Synthetic Data for AI Agent Evals: Build the Set Before You Have Users

Your agent has been live for 11 days. You have 40 real conversations, a Slack thread of screenshots from your only beta user, and a prompt change sitting in a draft PR that you are fairly sure fixes yesterday's bug. Neither number helps much yet: 40 real cases will not build a scored eval set, and waiting 6 months for 1,000 more is not a plan. Synthetic data for AI agent evals means using an LLM to generate plausible test inputs, and sometimes a graded expected outcome to go with each one, so you have something to score against before real traffic hands you enough cases on its own.

What is synthetic data for AI agent evals, and when do you actually need it?#

Synthetic data for AI agent evals is a set of test cases an LLM generates instead of ones pulled from real usage, built by seeding a generator with a handful of real examples and asking it for variations. You need it whenever real traffic is too thin to trust: before launch, right after launch, or for a rare edge case.

A production agent eventually gives you the real thing: actual phrasing, actual edge cases, actual failure patterns nobody predicted. The problem is the calendar. Wait for 1,000 real traces before building an eval set and you ship every change blind for months, finding out a prompt edit broke something only when a user hits it first.

Synthetic data does not replace that real data. It buys you a starting eval set immediately instead of after months of waiting, and it stays useful even once real traffic arrives, for the edge cases and adversarial inputs your limited user base has not thrown at you yet.

Three checks decide whether you can trust a synthetic eval set: a real seed, deliberate edge cases, and a paired expected outcome.

If you already have a few hundred real, varied traces, skip straight to building the eval set from those and treat synthetic generation as a smaller supplement for the gaps, not the foundation. The flowchart above is a gut check, not a rule: when in doubt, a blend of both beats waiting for either one to be perfect.

How do you turn a handful of real examples into a full eval set?#

The mechanism is seed, generate, pair, review. Start with even 5 to 10 real examples if that is all you have, ask an LLM to write variations and deliberate edge cases from that seed, attach an expected outcome to each new case, then have a human check the batch before it counts as part of your golden set.

This is not a new invention. It is the same seed-and-expand technique behind Self-Instruct and every "generate synthetic examples" button shipped since: give a generator a small set of real, labeled cases, and prompt it to produce more inputs that match the same shape and difficulty, not just the same topic. The seed set is what keeps the output grounded. Skip it and ask a model to write test cases for a support agent cold, and you get generic, easy inputs that miss what your actual users do.

Deliberately generating hard cases matters as much as generating more cases. Ask for edge cases and adversarial inputs by name in the prompt (an ambiguous request, conflicting instructions, a competitor's name dropped into the message, an empty or malformed field) rather than hoping normal variation produces them on its own. If your agent is RAG-grounded, the same technique works from source documents instead of chat logs: generate question, context, and answer triples straight from the documents it retrieves against, which is how our RAG evaluation metrics piece recommends building the labelled set its 4 scores run on.

Skip the seed step and the output drifts; skip the review step and you never find out.

Which tools actually handle this today?#

LangSmith, Promptfoo, and Braintrust each ship this as a built-in feature instead of a DIY prompt. LangSmith generates new dataset examples from ones you pick, Promptfoo expands seed prompts into test cases and separately red-teams for adversarial ones, and Braintrust generates cases through its Loop tool or pulls them straight from production traces.

You do not need any of these to start. A spreadsheet and a prompt does the job in an afternoon. But if your eval pipeline already lives in one of them, generation is a command or a button away, not a separate script to maintain.

Feature docs, if you want the specifics: LangSmith's synthetic example generator, Promptfoo's dataset and red-team commands, and Braintrust's dataset guide. Promptfoo's generator also takes a persona flag, so a batch can target several different user types in 1 run instead of 1 generic voice. Whichever you use, the review column matters more than the generate column. A tool that produces 200 synthetic cases you never look at is not an improvement over 20 real ones you did.

How do you pair a synthetic input with an expected outcome?#

Every eval case needs something to score against, and you have 2 honest options. A rule-based check works when the right answer is checkable by code: a status code, a required field, an exact string. An LLM-graded check works when correctness is fuzzier, like tone or completeness, and needs a rubric the same way a human grader would.

Rule-based grading is cheap and deterministic: the same input always scores the same way, which makes it the right default whenever the task allows it. Reserve LLM grading for cases where no rule can capture what "good" means, like whether a summary kept the important facts or a reply matched the requested tone.

The catch is that an LLM grader inherits every failure mode of an LLM judge, because that is exactly what it is. Non-determinism, leniency, and self-preference bias do not go away just because the thing being graded is synthetic instead of real. Our LLM-as-a-judge pitfalls piece covers all 6 failure modes in depth. The short version: a bad judge is one more way a synthetic eval set can lie to you, on top of any way the data itself is already wrong.

A hybrid works well in practice: use a rule-based check to catch the clear failures cheaply, and reserve the LLM grader for the harder middle cases a script cannot classify either way.

Default to rule-based wherever the task allows it; LLM grading inherits every LLM-as-judge bias.

Why does synthetic data drift from what real users actually do?#

Synthetic inputs drift from real ones in a measurable way: they run longer than what people actually type. A 2026 Airbnb study on synthetic search queries found real users write a median of 4 words, a generator seeded with real examples produces a median of 6, and one given no real seed averages nearly 14.

That is not a one-off finding. The same paper shows the gap is fixable: seeding the generator with real examples cut the distributional gap, measured by KL divergence against real query patterns, by roughly 7.5x compared to generating from scratch. The lesson is not that synthetic data is unreliable. It is that synthetic data without a real seed drifts, and synthetic data with one mostly does not. Source

Agent evals should expect a similar shape of gap even where nobody has measured it yet. A synthetic user message tends to read more complete and more polite than what a rushed real person types into a chat box, for the same reason a synthetic search query ran long in the Airbnb study: the generator defaults to a tidier, more grammatical version of the request. Treat that as something to test for, not a settled number.

That drift compounds if nobody checks for it. An eval set that quietly stops resembling real usage still produces scores, and those scores still look fine, right up until a real user hits an input your test set never imagined.

Every number here is read directly from the cited paper's own comparison of real versus synthetic search queries.

How does a model end up grading itself on evals it basically wrote?#

The failure mode is subtle: the same model family writes your synthetic test cases and grades the output, so both steps share the same stylistic habits. A 2025 study on self-bias in LLM-generated benchmarks found this compounds, with each model tending to rank itself first, even when a panel of other models disagrees.

This happens more easily than it sounds. Ask a model to generate synthetic test cases, then use a model from the same family to grade the outputs against them, and both steps draw on the same sense of what a good answer looks like. The model is not cheating on purpose. Its idea of a plausible edge case and its idea of a correct answer come from the same place, so the score it gives itself runs friendlier than a genuinely independent check would.

That is the same self-preference mechanism behind a biased LLM judge, just applied one step earlier, to the data instead of the score. Generating the test set and grading it with the same model stacks the bias instead of catching it. Source

The fix costs a second API call: generate synthetic cases with 1 model, grade with a different one, and keep a small slice of human-graded cases neither model ever sees during tuning. Treat the generator model as a real decision worth revisiting, not a default, especially anywhere the eval result decides which vendor gets paid.

Each check isolates one way the generator and the grader can quietly agree with each other.

What should you actually set up this week?#

Start smaller than feels responsible: 5 to 10 real examples, an LLM prompt that generates 20 variations and 5 deliberate edge cases from them, and a rule-based check wherever the task allows one. Review every generated case once before it counts. That is a working eval set by the end of the day, not a research project.

Once real traffic starts arriving, the job flips: fold every new real case into the set, weight it above the synthetic ones you started with, and keep the synthetic share around only for edge cases production has not surfaced yet. Langfuse's own golden-dataset guide suggests starting with 20 to 50 human-reviewed items and growing toward 100 to 1,000 for a full regression set. Source

If you have not built the harness that runs this set on a schedule, testing and evaluating AI agents in n8n covers the trigger-and-score loop this data feeds into. For the adversarial slice specifically, a red-team generator like Promptfoo's produces several hundred attack-style inputs in 1 run, more edge-case coverage than most teams would hand-write in a week.

A synthetic eval set is a start, not a substitute. Its whole job is to make itself obsolete as production data replaces it, section by section.