Model Distillation Isn't Using a Smaller Model. It's Training Your Own.

Your support-ticket classifier calls Opus 4.8 for every single ticket, and the monthly bill shows it: thousands of dollars to answer one narrow, repetitive question with a model built to do everything else too. Model distillation is the actual fix for that specific waste: you train a small model to do just this one job, using the frontier model's own outputs as its teacher, so production runs on the cheap model and the expensive one only gets called during training.

What is model distillation, and how is it different from picking a smaller model?#

Model distillation is training a new, small "student" model to imitate a larger "teacher" model's behavior on one specific task, using the teacher's own outputs as training data. It is a training process you run, not a purchasing decision. Swapping to an existing smaller model, by contrast, changes nothing about the model itself.

3 techniques get confused with each other constantly, and they solve different problems. Right-sizing, lever 1 in our guide to cutting an AI API bill, means routing a task to an existing cheaper model like Haiku 4.5 or GPT-5.6 Terra and checking whether it is good enough as-is. Generic fine-tuning means adapting a model, any size, on data you already have, usually human-labeled. Distillation is the specific case where the training data is generated by a larger model instead of a human, and the whole point is transferring one model's behavior into a smaller one.

Quantization and pruning are different again: they shrink an existing model's weights after training, cutting precision or removing parameters, rather than training a new model from a teacher's outputs. All 4 techniques can stack, but only one of them, distillation, actually teaches a small model something it did not already know.

How does the actual training mechanism work?#

The teacher model generates the training data for the student, either as full worked examples or as richer probability distributions over possible answers. Hinton et al.'s original 2015 method softens the teacher's output distribution with a temperature parameter, producing "soft labels" that carry more information than a single correct answer ever could.

In the classic version, the teacher does not just give a final answer, it gives a full probability distribution over every possible answer, softened by a temperature setting so the small differences between the second and third-best guesses become visible. That distribution is the soft label, and training the student to match it transfers more of the teacher's actual reasoning than training on the single correct answer alone.

Soft labels and rationales carry more of the teacher's actual reasoning than a single correct answer.

Most builders never touch raw logits, though. Hosted APIs from OpenAI, Anthropic, and Google do not expose full output distributions, so practical LLM distillation usually works on text: the teacher writes out its full answer, sometimes with its reasoning shown, and the student is fine-tuned to reproduce that. Google Research's "Distilling step-by-step" method pushes this further, extracting the teacher's rationale, not just its final answer, as extra supervision. Their tests found a fine-tuned 770M-parameter T5 model beat a 540B-parameter PaLM model prompted few-shot, using only 80% of the available training data. Source

What does the distillation pipeline actually look like, step by step?#

A distillation pipeline runs in roughly 6 stages: define the task narrowly, pick a teacher model, generate training examples from it, choose a student model size, fine-tune the student on those examples, then evaluate it against both the teacher and the undistilled student before deploying. Skipping the narrow-task step is the most common way the whole project fails.

Skipping the first step is the most common way the whole project fails.

The task-definition step does the most damage when skipped. "Handle customer support" is too broad to distill well; "classify this ticket into one of 8 categories" or "extract the order number and complaint type from this email" is narrow enough that a small model can actually learn the pattern. Dataset size varies by task, but it is smaller than most builders expect: Amazon Bedrock's distillation jobs top out at 15,000 prompt-response pairs even when its data-synthesis step generates extra variety on top of what you provide. Source Student size is a similar judgment call: a model in the 1B to 8B parameter range is the common starting point for a single narrow task, sized up only if evaluation shows it is actually short of capacity, not on a guess.

Evaluation itself needs more than a single accuracy number to be trustworthy. Track exact-match or accuracy against a held-out set the student never trained on, watch latency and cost per call since those are the entire point, and keep a small sample for manual review, because a distilled model can hit a good aggregate score while still failing in a specific, embarrassing way an automated metric misses.

What do OpenAI, AWS, and Google actually ship for this today?#

OpenAI, Amazon, and Google each ship a first-party distillation feature, though at different levels of maturity. OpenAI's version fine-tunes a cheap model on stored outputs from a frontier one; AWS Bedrock automates teacher-to-student distillation for models like Claude; Google's is still in early access and explicitly not for production use yet.

OpenAI's approach lives inside the API platform itself. Set store: true on a Chat Completions call to a model like gpt-4o and OpenAI saves the input-output pairs; filter those stored completions down to the ones you want, click "Distill," and it opens a fine-tuning job against a cheaper base model like gpt-4o-mini. OpenAI's own cookbook example is a genuinely useful data point: on their internal benchmark, the undistilled gpt-4o-mini scored 69.00% against teacher gpt-4o's 81.80%, and the distilled version closed most of that gap at 79.33%, a jump OpenAI itself describes as roughly a 22% relative improvement. Source

Every number here is read directly from the vendor's own documentation, not a summary of a summary.

Amazon Bedrock Model Distillation, generally available since May 2025, automates the same shape of workflow for models including Claude: you pick a teacher and a student, Bedrock generates synthetic responses from the teacher, or reuses real ones from your invocation logs, and fine-tunes the student on the result. You pay for the teacher's on-demand inference during data generation, plus the fine-tuning and hosting cost for the student. Source The pattern's earliest public proof point is from December 2024, when Anthropic and AWS first previewed it: Claude 3 Haiku, distilled from Claude 3.5 Sonnet, reached "Claude 3.5 Sonnet-like accuracy for specific tasks" while keeping Haiku's price and speed. Source Treat that as an early proof of the mechanism, not a live 2026 benchmark.

Google has opened a similar Gemini Distillation Service, but it is explicitly early access: Google's own guidance describes models distilled during this phase as for experimentation only, not for production, and access requires requesting an allowlist. Worth watching, not yet worth building on.

None of these 3 remove the core work: you still have to define the task narrowly and check the result actually holds up before you trust it in production.

When is distillation worth the setup effort, and when isn't it?#

Distillation pays off when a task is narrow, high-volume, and stable enough that a small model can actually learn its shape, and when you have or can generate enough representative examples to train on. It does not pay off for broad or shifting tasks, low-volume calls, or anything where the model needs judgment it was never shown during training.

Every pro here is a real mechanism; every con is a real cost. Neither cancels the other out.

The tasks that distill cleanest share a shape: a fixed, small set of possible outputs, like ticket routing or sentiment labels, or a tightly-scoped extraction like pulling a date and an amount from an invoice. The tasks that distill worst are open-ended ones where the correct answer is really a judgment call, general customer chat, creative writing, anything where two competent humans would give different answers, because the student has nothing consistent to learn.

The honest failure mode is brittleness. A distilled model is only as good as the examples it learned from, and it degrades hard the moment a real request falls outside that distribution: a phrasing it never saw, an edge case the teacher never generated, a task it gets quietly asked to stretch into. That is exactly the problem a routing gateway is built to catch: keep the frontier model as a fallback, route the routine, in-distribution share of traffic to your distilled model, and let the gateway send anything unfamiliar back to the model that can actually handle it. Distillation without that safety net is a bet that your training examples covered the real traffic; with it, a wrong bet just costs one expensive call instead of a bad answer in production.

Should you distill your own small model for this task?#

4 questions in sequence settle it: is the task narrow and well-defined, is call volume high enough that frontier cost or latency actually hurts, does an existing smaller model already clear the bar, and do you have enough teacher-quality examples to train on. A "no" at any single step points to a cheaper alternative than building a distillation pipeline.

Four questions, in order: task scope, call volume, an existing stock model, and enough teacher examples to train on.

Walk the ladder honestly and most tasks resolve in under a minute: either you already have a cheaper fix on hand, right-sizing among them, or you have a real case for training a student model.

What should you set up this week?#

Start smaller than a full pipeline: pick one narrow, high-volume task, generate 100 to 200 examples with your frontier model, and manually check whether a distilled model would even have room to improve versus an existing cheap model. That one afternoon of prep tells you whether the 4-question ladder above is worth walking for real.

That prep work also tells you what you are signing up for operationally. A distilled model is a model you now own: you version it, you re-evaluate it when the task drifts, and you decide where it runs, which puts you back in the open-weight versus closed deployment decision, just with a model nobody else has trained. Most teams that get real value from distillation are not running dozens of student models. They found one narrow, expensive, high-volume task, proved a small model could learn it, and stopped there.