Choosing an LLM for Production: The Decisions That Stick

Every model comparison you read was written against someone else's task. The only ranking that matters is the one your own evaluation produces, and building that evaluation is cheaper than the month you will otherwise spend switching models on vibes.

The symptom is familiar. A new model ships, the benchmarks look better, you swap it in, something subtle degrades three weeks later, and you cannot tell whether it was the model, the prompt, or a change upstream, because you never had a number.

What you will end up with#

A written quality bar, a thirty-case evaluation, a per-job model choice with a cost you can defend, and a swap that takes minutes rather than a sprint. Twelve lessons, each one a decision that is expensive to reverse later.

What this does not cover: training or fine-tuning your own model, and anything that needs a GPU you have to rent.

Every stage depends on the bar written in the first one.

Lesson 1Define good enough, in writing, first#

Before you open a single comparison, write down what your task requires. Not "high quality". A behaviour, stated so that a run either passes or fails.

The shape that works:

Task     : classify inbound support email into 6 categories
Passes if: correct category, and "unsure" rather than a guess when ambiguous
Fails if : invents a category, or guesses confidently on an ambiguous case
Bar      : 90% correct on 30 real emails, ZERO confident wrong answers

That last line does the work. Most tasks have one failure that costs far more than the others, and naming it turns a fuzzy preference into a gate. A model that scores 94% with two confident wrong answers loses to one that scores 91% with none.

There is a second reason to write it down. Six weeks from now a new model ships, someone asks whether you should switch, and the bar is the only thing that turns that into a five-minute question instead of an afternoon of opinion.

1. Name the task in one sentence.
2. Name the one failure you cannot tolerate.
3. Pick a number you would accept, before you know what any model scores.

Point 3 matters because a bar chosen after seeing results is not a bar, it is a rationalisation.

Aside 1.5The benchmark trap#

Public benchmarks measure a model's average across tasks that are not yours. They are useful for ruling models out and nearly useless for ruling one in.

Three specific reasons, all of which bite:

There is a fourth reason that gets less attention: contamination. Popular benchmark sets leak into training data over time, so a score can improve without the underlying capability improving. That is not an accusation about any particular model, it is a structural property of publishing a fixed test set, and it is another reason your own thirty cases are worth more than a public number.

The practical use of a leaderboard is to pick three candidates to test, not to pick one to ship. Treat it as a shortlist generator.

The current field, with what each tier is genuinely for, is in The Frontier Model Field in 2026.

Lesson 2Start at the cheapest tier that could plausibly work#

Start low and move up only when your evaluation forces it. The reverse order feels safer and costs far more, because a flagship model that works tells you nothing about whether a cheap one would have.

The spread is larger than most people expect. On identical workflow logic, 1,000 runs came to $3.30 on a fast tier against $81.00 on a flagship, a 25x difference for the same work. If the cheap tier passes your bar from lesson 1, the expensive one is not better, it is just more expensive.

Most production work lives at the base. Move up only when the eval forces it.

Read the failure column rather than the use column. A tier is not defined by what it can do on a good day, it is defined by how it fails on a bad one, and those failure modes are what your evaluation from lesson 3 is designed to surface.

The practical move: run the fast tier first even when you are fairly sure it will not pass. If it does pass, you have saved 25x. If it fails, you now know exactly which case it failed on, which is a far better brief for the next tier up than "it felt weak".

The measured comparison inside a real workflow is in Claude vs GPT vs Gemini in n8n, and the per-job framing is in Frontier Model Default Per Job.

Lesson 3Build the evaluation before you compare anything#

Thirty real inputs with expected outcomes, run the same way every time. This is the hour that pays for itself more than any other in the process and it is the one people skip, because it feels like preparation rather than progress.

1. Take 30 real inputs from your actual traffic, including the three weirdest.
2. Write the expected OUTCOME for each, not the expected wording.
3. Include at least five cases that should fail, refuse, or escalate.
4. Score automatically, so re-running costs nothing.

Step 3 is what makes the set discriminating. A set of only happy paths gives every candidate a near-perfect score and no signal.

Two practical notes on building the set. Take the inputs from real traffic rather than writing them, because invented inputs are unconsciously written to be answerable and yours will be too clean. And keep the set small enough that re-running it is free in both time and money; thirty cases you actually re-run beats three hundred you run once and then avoid.

The full method, including scoring and mocking, is in How to Test and Evaluate n8n AI Agents. If you do not have 30 real inputs yet, Synthetic Data for Agent Evals covers generating a starting set honestly.

Lesson 4Compare on your task, not the leaderboard#

Run the same evaluation against three candidates: one fast tier, one mid tier, and whatever the current default is. Same prompt, same inputs, same scoring.

The output you want is a small table you can defend to yourself in six weeks:

Read that table the way it deserves. The mid tier scored highest on raw passes and still loses, because it produced the one failure the bar in lesson 1 said was unacceptable. The fast tier is three points behind on a metric that does not matter and clean on the one that does.

One more column is worth adding once you have run this twice: variance across runs. A model that scores 29, 29, 29 is a different proposition from one that scores 30, 29, 27, even though the second has a better peak. Consistency is a feature you are buying, and it does not appear on any leaderboard.

Aside 4.5The A/B that actually tells you something#

Two mistakes make a comparison meaningless, and both are easy to make.

Changing the prompt between models. A prompt tuned for one model's tool-calling habits is not a neutral test of another. Run the same prompt first, note where each fails, then tune per model in a second round if you care enough.

Testing once. Models are not deterministic. A single run of 30 cases has real variance, and a two-point difference between two candidates is noise. Run the set three times and compare the worst run, not the average, because production will find the worst run for you.

for model in fast mid flagship:
  for run in 1..3:
    score = evaluate(model, golden_set)
  report worst(score), not mean(score)

Comparing worst runs is pessimistic on purpose. The question is not how good a model can be, it is how bad it gets when you are not watching.

Lesson 5Price it before you commit, not after#

Token pricing is per million and your intuition about it is wrong, because the number that matters is not the price, it is the price times how the tokens actually accumulate.

The last row is the common misdiagnosis. Output tokens cost more each, so people optimise the answer length first, while the input side is ten times larger and re-sent every pass.

Two things that calculator makes visible. Input dominates output on most real workloads, because prompts, tool schemas and context are large while answers are short. And anything that loops multiplies the input side, not the output side.

The token arithmetic in full is in Estimate LLM API Cost.

Aside 5.5The costs that are not per-token#

Four line items sit outside the price-per-million and each one has caught people out.

1. Cached input reads at a fraction of the fresh price, with a one-time write premium. For anything with a stable prefix this is the largest single lever available.
2. Batch processing is roughly half price on several providers, if your work tolerates a delay.
3. Failed and retried calls are billed. A flaky tool that triggers three retries triples that run.
4. The long tail of long inputs. Average input size is a bad planning number; the p95 is what breaks the budget.

Point 4 is the one that surprises people. Costs are driven by the biggest inputs, not the typical ones, and averages hide them completely.

The caching mechanics are in Prompt Caching to Cut LLM Cost and the batch discount is in The LLM Batch API 50% Discount.

Lesson 6Treat latency as a separate decision#

Quality and latency are different axes and the fastest model is often not the cheapest. Decide which one your task actually needs before you trade anything away.

Measure the p95, not the mean. Model latency has a long tail, and the slow requests are the ones your users notice and remember. A model with a good median and an ugly tail reads as unreliable even when the average looks fine.

Streaming changes the perceived number more than switching models does: a slower model that starts producing in 300ms feels faster than a quicker one that stays silent for two seconds.

The streaming and perception detail is in LLM Streaming Response Latency and UX, and the agent-loop version is in n8n AI Agent Slow Latency and Cost.

Lesson 7Make the choice reversible on day one#

Every decision above has a shelf life measured in weeks. Wire the model as one swappable reference from the start, so re-deciding is a config change rather than a project.

1. One place in the codebase or the canvas names the model.
2. The prompt carries no provider-specific syntax.
3. The evaluation from lesson 3 runs on demand, so a swap can be checked in minutes.

In n8n that is literally one sub-node under the agent. In code it is one function that everything else calls.

Aside 7.5The compatibility layer that makes swapping trivial#

Most providers now expose an OpenAI-compatible endpoint, which means a swap can be a base URL and a model name rather than a rewrite.

base_url = https://api.provider-a.com/v1     model = fast-tier-a
base_url = https://api.provider-b.com/v1     model = fast-tier-b

A gateway is worth its complexity at roughly the point where you are running three or more providers, or where you need spend visibility per feature rather than per account. Below that it is a dependency you are adding to solve a problem you do not have yet.

The caveat worth knowing: compatible does not mean identical. Tool-calling formats, streaming chunk shapes and error codes drift between implementations, so re-run the evaluation after any swap rather than assuming parity.

The details are in OpenAI-Compatible API Model Swap and the routing layer is covered in LLM Model Routing and Gateways.

A related trap: the escape hatch only works if you have used it. A second provider that is wired but has never served a real request is a plan, not a fallback. Send it a small share of live traffic occasionally, or run the eval against it on the same quarterly cadence, so that the day you need it is not the day you find out the tool-calling format differs.

Lesson 8Decide open-weight against closed deliberately#

This is a constraint decision, not a quality one, and it should be settled by your requirements rather than by preference.

A constraint decision, not a quality one. Let the requirements settle it.

The gap between the best open-weight models and the frontier has narrowed considerably, which moves this decision for a lot of workloads that were closed-only a year ago.

The current comparison is in Open-Weight vs Closed LLMs, and the cost crossover for running locally is in Local LLM vs API: Cost Breakeven.

Lesson 9Route per job, not per application#

"Which model should I use" is the wrong shape of question. A real application has several jobs with different requirements, and forcing one model onto all of them overpays for the easy ones and underserves the hard one.

Forcing one model on every job overpays for the easy ones and underserves the hard one.

The routing itself does not need to be clever. A switch on the job name, set where the workflow already knows what it is doing, covers almost every real case. Adaptive routing that picks a model per request based on predicted difficulty is a real technique and it is almost never where the money is: the win comes from not sending classification work to a flagship, and a static rule does that.

That last row is the important one. Reaching for a bigger context window is frequently the expensive answer to a question that retrieval solves more cheaply and more accurately.

The retrieval alternative is in Contextual Retrieval for RAG, and the context-window mechanics are in LLM Context Windows Explained.

One caution on routing: every branch you add is a path that needs its own evaluation. Three jobs routed to three tiers means three sets of cases, or you are only measuring one third of your system. This is the real cost of routing and it is why "one model everywhere" survives longer than it should in most codebases.

Lesson 10Plan for the model changing under you#

A hosted model is not a fixed dependency. Versions are deprecated, defaults move, and behaviour shifts within the same name. If your quality depends on a model you do not control, that dependency needs managing like any other.

1. Pin an explicit model version wherever the provider allows it. Never ship pointing at "latest".
2. Re-run the evaluation on a schedule, not only when something looks wrong.
3. Keep a second provider wired and tested, from lesson 7.
4. Subscribe to the provider's deprecation notices, and treat them as dated work.

Aside 10.5The deprecation that will happen#

Every provider retires models, usually with a date and usually with less notice than you would like. The teams this hurts are the ones who find out from an error rather than from a calendar.

The pattern that works is unexciting:

when a deprecation lands:
  1. run the eval against the successor      <- do this the week you hear
  2. note the delta, especially any NEW confident-wrong cases
  3. swap in a low-traffic window, keep the old version pinned until the deadline
  4. re-run the eval a week later on real traffic shape

Step 4 is the one people skip. Your evaluation is thirty cases; production is thousands, and the difference shows up in the tail.

Two real migrations, written up with what actually broke, are in Gemini API Migration Playbook and DeepSeek Model Migration.

Lesson 11Cut the bill without changing the model#

Before you downgrade on quality, take the free money. Four levers reduce cost without touching which model you chose.

1. Cache the stable prefix. System prompt, tool schemas, long shared context. The repeated part of a prompt is usually the large part.
2. Shorten the input, not the model. Trim retrieved context and memory windows. Most prompts carry material the task does not use.
3. Batch anything that can wait. Roughly half price where the provider offers it.
4. Cache the whole answer for repeated questions. A semantic cache costs nothing when it hits.

All four cut cost without trading quality. A cheaper tier does.

The order matters more than the individual items. Teams reach for a cheaper model first because it is the most visible lever, and then discover they gave up quality to save less than caching would have saved for free.

The first four cost nothing in quality. Only the last one trades it.

Only after those four should you consider a cheaper tier, because that is the lever that trades quality, and the other four do not.

The semantic-cache pattern is in LLM Semantic Cache, and the broader list is in Reduce AI API Costs: 7 Levers.

Lesson 12Re-run the decision on a schedule#

The right model six months ago is not the right model now, and nothing tells you when it changed. Put it on a calendar rather than waiting for a reason.

Quarterly is enough for most workloads:

1. Re-run the evaluation against your current model, to catch silent drift.
2. Add whatever shipped since as a fresh candidate.
3. Re-check the price. Tiers get cheaper more often than they get better.
4. Change nothing unless the numbers say so.

There is one trigger that should jump the queue rather than wait for the quarter: a provider deprecation notice with a date on it. Those arrive on the provider's schedule, not yours, and the work is the same either way. Doing it in the week you hear costs an hour; doing it in the week of the deadline costs whatever else you had planned.

Point 4 is the discipline that makes the rest sustainable. The purpose of the review is a decision you can defend, and "no change" is a perfectly good outcome that you now have evidence for.

The recurring read on what shipped is in July 2026 Model Wave for Builders.

What I would do differently#

Build the evaluation before comparing anything. The failure mode is comparing first, choosing on an impression, and being unable to reconstruct the reasoning a month later, which is what happens every time the test set feels like the boring part.

It is the boring part. It is also the only part that produces an answer rather than an impression.

The second thing is tier discipline. The flagship gets defaulted to on anything that feels important, and "feels important" usually means "makes me nervous", which is not a technical criterion. Most of those jobs are classification and extraction, which the fast tier handles at a fraction of the price, and the difference gets paid for months in exchange for feeling safer.

Third, smaller but sharper: pin the version. When a default moves underneath a workflow the output changes subtly, and the instinct is to look at the prompt first. That is a day gone to something a single line prevents.

One note on ordering. The four free levers in lesson 11 come before any tier change, because they cut cost without touching quality and a cheaper model does not. Teams reach for the model first because it is the most visible lever, then discover they traded quality to save less than caching would have saved for nothing.

What I am still unsure about is how often to re-run the decision. Quarterly is what I do, and it is a guess dressed as a policy. The tiers have been getting cheaper faster than they have been getting better, which argues for checking price more often than capability, but I have not found a cadence I can defend with anything more than "it has not burned me yet".

Changing nothing is a valid outcome, and now it is an evidenced one.

Where to go next#

If you have no evaluation yet, lesson 3 is the only place to start; every other lesson depends on having a number.

If you have a model running and the question is cost, start at Reduce AI API Costs. If the question is whether to run it yourself, Local LLM vs API: Cost Breakeven has the crossover. And if the model is going inside an agent loop rather than a single call, the cost shape changes enough that How to Build an AI Agent covers it from the other side.