Serverless vs Dedicated GPU: The Inference Cost Breakeven

Choosing between serverless GPU vs dedicated GPU inference cost is not a preference, it is arithmetic with one unknown. The unknown is your duty cycle, and almost nobody measures it before signing for a month of reserved capacity. Get that number and the decision makes itself, usually against whichever option you were already leaning toward.

What actually decides which one is cheaper?#

Utilization, meaning the fraction of wall clock time your GPU is genuinely computing. Dedicated capacity charges for every second you hold the card. Serverless charges only for seconds of work but at a higher rate per second. Above some duty cycle the always-on card wins, and below it you are paying rent on idle silicon.

The formula is short enough to do in your head. Let k be the serverless rate divided by the dedicated rate for the same class of card, per unit of compute time. Dedicated becomes cheaper once your utilization exceeds 1 divided by k. If serverless costs twice as much per second, the crossover sits at 50 percent utilization.

Two things make that more useful than a benchmark someone else ran. It uses your provider's current rates rather than a table that goes stale, and it forces you to state a utilization figure out loud, which is where most of the disagreement actually lives.

The first row explains every other row. One bills for work done, the other for capacity held.

Measure utilization rather than estimating it. The intuition is reliably wrong because people think in requests per day and the card thinks in seconds of occupancy. A service handling 5,000 requests a day at 200ms each is busy for under 17 minutes in 24 hours, which is roughly 1 percent.

Why is idle time so much larger than people expect?#

Because traffic is bursty and GPUs are not shared the way web servers are. A card serving a business-hours product is doing nothing for two thirds of the day before you count weekends, and it is doing nothing between requests during the busy hours too. Reserved capacity charges for all of it.

Batching changes this and is worth doing regardless of which option you pick. Serving several requests concurrently raises occupancy without adding cards, so the same hardware absorbs more traffic and the utilization figure in the formula moves in your favour. It also makes dedicated capacity look better, since you are filling what you already rent.

Modeled at k equal to 2, so the crossover sits at 50 percent. Recompute with your own provider's k.

The pattern that catches teams is growth. A workload at 8 percent utilization is obviously serverless, and the same workload at 55 percent after a good quarter is obviously not, and nobody re-runs the calculation because the first answer felt settled. Put a date on it.

What does a cold start actually cost you?#

Tens of seconds, and it is the real price of scale to zero. A cold serverless GPU has to schedule a container, initialise the framework, pull multi-gigabyte weights into memory and warm up CUDA graphs. Reported figures for that sequence run from about 30 to 60 seconds, with production setups exceeding 40 seconds to a first token.

The components are worth seeing separately because they respond to different fixes. Weight loading is bandwidth bound, graph capture is compute bound, and container scheduling is queue bound. Only the first is helped by a faster disk, which is where most optimisation effort goes by default.

Set against those figures, a 40 second first token is not an edge case, it is the expected behaviour of a genuinely cold card. Systems that keep a warm pool or use tiered checkpoint loading report large reductions, but a warm pool is dedicated capacity wearing a serverless label, and it should be costed as such.

Which workloads suit which option?#

Batch and spiky interactive work suit serverless; steady interactive work suits dedicated. If a job runs nightly for 40 minutes, paying for 24 hours of card is indefensible. If a product answers users continuously through the working day, cold starts are a user-visible defect and reserved capacity is the cheaper way to avoid them.

Latency tolerance is the second axis and it often overrules cost. A voice interface cannot absorb 40 seconds under any billing model, so the choice collapses to dedicated or to a warm pool that costs like dedicated. An overnight document pipeline can absorb minutes without anyone noticing, which makes scale to zero close to free.

Every item here has moved a real decision. None of them appear on the page you compared rates on.

The hybrid is legitimate and under-used. Hold a small dedicated floor for the traffic you always have, and let serverless absorb the peaks above it. That prices the predictable part at the cheaper rate and the unpredictable part at the flexible one, which is what both billing models are actually good at.

How do you work out your own crossover?#

Pull 4 weeks of request logs, multiply the request count by mean compute seconds, and divide by the seconds in the period. That is your utilization. Then take the ratio of the two rates your provider quotes, invert it, and compare. Two numbers, 10 minutes, and the answer holds until your traffic shape changes.

Two numbers decide it. The rest of the argument is usually about which number nobody measured.

Include queueing in the compute seconds if requests wait. A card that is occupied but stalled is still rented, and a workload that looks like 30 percent utilization on paper can be considerably higher in practice once concurrency limits and retries are counted. Concurrency and rate limits shape this more than model choice does.

When is this the wrong question entirely?#

When a hosted API would serve the workload at less than either option. Self-hosting earns its keep on volume, on data residency, or on a model nobody hosts for you, and at low volume it usually loses to a metered endpoint by a wide margin once engineering time is counted.

The hosted API question comes first and removes this decision more often than people expect.

The honest sequence is to check the hosted option first, then compare serverless against dedicated only if self-hosting already made sense for a reason other than price. Working out the self-hosting breakeven answers that prior question, and it eliminates this one more often than people expect.