4 Signals Decide Whether a Pipeline Step Needs a Frontier Model

Your support bot calls the same frontier model to classify a ticket's category and to draft the reply that goes out to the customer, because that is the one model your code already knows how to call. The classification step gets the label right either way, at a price and a latency built for the far harder job that happens later in the same pipeline. Small versus large is not a decision you make once for the whole agent, it is a decision you make once per step, and most pipelines have never had that audit run on them.

What actually makes a pipeline step a candidate for a small model?#

An agent pipeline is rarely one model doing one job. It is a chain of steps, classify, retrieve, plan, generate, each with its own difficulty. A step is a small-model candidate when its job is narrow and repeatable enough that a cheaper model can hit the same practical accuracy as a frontier one, on that step alone.

Small now covers 3 different sourcing paths, and the decision framework below applies to all of them the same way. You can distill your own model from a frontier teacher for a task nothing off-the-shelf covers well. You can pull a general-purpose small model, something in the Phi, Gemma, or Qwen families, off the shelf and run it on a server or on the device itself. Or you can just call a frontier vendor's own cheap tier, Claude's Haiku 4.5 or OpenAI's GPT-5.6 Luna, which is a small model in every sense that matters here even though nobody markets it that way.

None of those 3 paths changes the question this post answers: is this step, the one you are looking at right now, narrow and cheap to verify enough that the smaller option is not a downgrade. NVIDIA researchers made a version of this argument directly in a 2025 position paper, pointing out that agentic systems ask a model to do a small number of specialized, repetitive things far more often than they ask for open-ended conversation, and estimating that serving those repetitive calls on a small model runs roughly 10 to 30 times cheaper than defaulting to a frontier one. Source Read that as the paper's own framing, not a settled industry number, it is one research group's position, not a benchmark every vendor has signed off on.

Neither model wins every row. The row that matters depends on the step, not on the agent as a whole.

Is the output space narrow, or genuinely open-ended?#

The first signal is the shape of the output. Classification into a fixed set of labels, extracting a date or an amount from text, routing a request to the right queue, these all have a small, countable set of correct answers. A small model can learn that shape well. Open-ended generation has no such ceiling to learn.

A ticket classifier picking from 8 categories, a form parser pulling a name and an order number, a router deciding which of 3 downstream tools to call, all of these have an answer key you could write down in a spreadsheet before you ever call a model. That is the tell. If you can enumerate the plausible outputs, or at least bound them tightly, a small model has something concrete to learn and something concrete to be graded against.

Drafting the reply itself, planning a multi-step task, summarizing a document nobody has seen before, none of these have a fixed answer key. Two competent people would write different, both-correct versions. That is not a small-model problem because it is hard, it is a small-model problem because there is no narrow target to hit in the first place.

Does the task need multi-hop reasoning, or one clean judgment call?#

The second signal is how many steps of reasoning the task actually takes. A single-hop judgment, is this spam, does this match that category, reads well on a small model. A task that has to hold several facts in mind, weigh them against each other, and revise a plan mid-stream is where small models tend to fall apart fastest.

A 2026 study on model routing puts real numbers on that gap, on a task about as small-model-friendly as it gets. Researchers benchmarked a 3-billion-parameter model, Qwen-2.5-3B, against a much larger one, DeepSeek-V3, sorting a request into 1 of 6 categories before deciding which downstream model should answer it. The small model ran at 988 milliseconds and $0 marginal cost with 79.3% accuracy; the large model scored 83.0% but blew past the study's own latency limit. That is a 3.7 point accuracy gap in the large model's favor, and neither one cleared the 85% bar the researchers had set for calling a router production-ready on its own. Source Treat this as one modest study and not a verdict on every classification task, its own test set ran about 60 cases per arm, but it is a rare SLM-versus-LLM comparison with real, checkable numbers attached instead of a vendor's marketing claim.

The practical read: on a genuinely narrow, single-hop task, the small model landed within about 4 points of a vastly larger one, at a fraction of the latency and cost. That is close enough to be worth testing on your own task. It is not automatically close enough to skip testing altogether.

Can you measure whether the step got it right, and does a miss cost you?#

The third signal is not about the task, it is about you: can you actually tell when the step is wrong. A step you can grade against a held-out set, cheaply and often, is safe to hand to a cheaper model, because drift shows up fast. A step you can only judge by reading it yourself is not.

Measurability and stakes travel together in practice. A classification step is both narrow and cheap to check, you compare the label against ground truth and get a number back in seconds. A step that drafts a legal clause or approves a refund is a different animal even if its output space looks bounded on paper, because a wrong answer there is expensive in a way a wrong ticket-routing label is not.

The honest question to ask per step is not "is this task hard" but "what happens when the cheaper model is wrong, and how fast will I find out." A miss that gets caught by the next step in the pipeline, or by a person before it reaches a customer, is cheap. A miss that ships straight to production unreviewed is not, and that alone is a reason to keep a step on the frontier model even when its output space is narrow.

Clearing most of these is the actual test, not a general sense of how hard the task feels.

When does the cost and latency math actually start to matter?#

The fourth signal is pure economics: volume. A step called 5 times a day saves you pocket change no matter which model runs it, so the setup cost of testing and swapping is not worth it yet. A step called thousands of times a day turns even a small per-call saving into a real number on the monthly bill.

Put real numbers on it. Claude's Haiku 4.5 lists at $1 in and $5 out per million tokens, against Opus 4.8's $5 and $25, a 5x gap on both sides. Source Route a high-volume classification step from Opus to Haiku and the bill on that step drops by roughly 80%, before you even touch caching or batching. Our cost hub, 7 levers that cut an AI API bill, calls this right-sizing and names it the single biggest lever for a reason: it needs no new infrastructure, just a config change and a test set.

Low-volume steps are where this signal argues the other way. A step called a dozen times a week is never going to show up on the bill no matter what model answers it, so the engineering time spent testing and swapping a cheaper model there is a net loss. Save that effort for the steps your logs show running constantly.

Every number here traces to a cited paper or a published pricing page, not a summary of a summary.

What does this look like across a real pipeline?#

Take a common shape: an agent that classifies a user's intent, retrieves relevant context, then generates a final answer. Run all 3 signals against each step and the picture splits cleanly. Classification is narrow and single-hop. Retrieval is often not even a model call. Generation is the one step that stays open-ended, and stays expensive.

The split is per step, and it can move as a step gets better defined.

Classifying intent has a fixed label set, one hop of reasoning, and an answer key you can build from a week of support tickets, every signal points the same way. Retrieval usually is not a language model call at all, it is an embedding lookup against a vector index, so the question barely applies. Generation is where the signals flip: the output is open-ended, a bad answer reaches a customer directly, and that is exactly the step worth protecting with a frontier model even after the first 2 steps move to something cheaper.

Nothing about this split is permanent. A team that gets serious about the generation step might eventually narrow it too, a constrained output format, a template with blanks to fill, and pull it down a tier later. The point of running the signals per step is that the split can move in either direction as a task gets better defined, not that generation is forever exempt.

Should you route this step to a small model right now?#

Run the 4 signals as a short ladder instead of a single verdict: narrow output space, single-hop reasoning, measurable accuracy, enough volume to matter. Clear every rung and the step is a strong small-model candidate. Stall on one of them and the step stays on the frontier model until that specific blocker changes.

Four gates, in order: output shape, reasoning depth, measurability, and volume.

Once you have sorted a pipeline's steps this way, the mechanical part is a routing decision at call time, not a research project. A model routing gateway is the natural place to enforce the split: point the classify and retrieve steps at the small model by default, keep generate pinned to the frontier model, and let the gateway's fallback catch anything the small model returns with low confidence or an out-of-taxonomy label.

What should you set up this week?#

Pick one pipeline you already run and score each step against the 4 signals: output shape, reasoning depth, measurability, and volume. You will likely find one obvious candidate, often the classifier or the router, that has been running on a frontier model out of habit rather than need. Swap that one step first.

Run the comparison for real: same input, both models, side by side, graded against the same held-out set. If the small model matches within a margin you can live with, at a fraction of the cost and latency, move it and keep the frontier model as a fallback behind a gateway rather than deleting the option entirely.

The steps that stay on a frontier model are not a failure of this exercise, they are the correct answer for that step. An agent pipeline built entirely on one model size was never the efficient design, it was just the easiest one to ship first. Auditing it step by step is the actual upgrade, not any single model swap.