Skip to content
TheAgent Ecosystem
Models & Cost

GPU Sizing for a Self-Hosted LLM: How Much VRAM?

The arithmetic behind the answer, so it still works on models that do not exist yet.

Muhammad Qasim HammadAI-assisted9 min read1,856 words

AI-drafted, reviewed by Muhammad Qasim Hammad on November 9, 2026. See our AI disclosure.

SELF-HOSTED LLM: How Much VRAM Do You Need
Table of contents
  1. How much VRAM does a self-hosted LLM need?
  2. Why does a model that fits still run out of memory?
  3. What happens when it almost fits?
  4. How do you size it in five steps?
  5. What does quantization actually cost you?
  6. What breaks the calculation?
  7. Should you rent or buy?

You have done the break-even math and self-hosting looks worth it. Now you have to buy or rent something, and every guide answers with a table of model names against card names that was already out of date when it was published. GPU sizing for a self-hosted LLM comes down to one piece of arithmetic you can apply to any model, including ones that do not exist yet.

How much VRAM does a self-hosted LLM need?#

Weights first: parameters times bytes per parameter. Precision sets the bytes, so FP16 costs 2 per parameter, INT8 costs 1, and INT4 about 0.5. A 7B model is therefore roughly 14GB at full precision and roughly 3.5GB quantized to four bits. That single relationship answers most of the question.

The reason this beats a lookup table is that it keeps working. New models arrive constantly, cards get replaced, and the arithmetic does not care. If you know the parameter count and the precision, you know the resting size.

Two conventions are worth knowing so the numbers stop looking arbitrary. Parameter counts are quoted in billions, so a 7B model is seven billion parameters, and the memory figure lands in gigabytes because a billion parameters at two bytes each is two gigabytes. And precision names describe bits per number: FP16 is sixteen bits, which is two bytes. Once those two click, you can do this in your head.

Table of approximate weight memory for 7B, 13B, 32B and 70B parameter models at FP16, INT8 and INT4 precisionWeights only. The KV cache is extra and grows with context length, so treat every figure here as a floor rather than a budget.

One family of models breaks this arithmetic, and it is increasingly common. A mixture-of-experts model activates only a fraction of its parameters per token, so it computes like a small model while still needing the whole thing resident in memory. A model advertised as fast because only a few billion parameters are active still occupies memory for its full parameter count. Size against the total, never the active count, or you will buy a card that cannot load the model at all.

What the table above does not include is the part that actually decides whether your setup survives a real conversation.

Why does a model that fits still run out of memory?#

Because weights are only the resting size. Every token in the context window keeps a key and value entry in memory, and that KV cache grows with conversation length. A model that loads comfortably at 4k context can exhaust the same card at 32k, which looks like a crash rather than a sizing mistake.

This is the single most common reason a self-hosted setup fails in week two rather than on day one. The model loaded, the first few requests worked, and then somebody had a long conversation.

The cache scales with context length, layer count, and how the model handles attention heads. Modern architectures using grouped-query attention are dramatically cheaper here than older ones, which is why a blanket formula is worth less than a measurement on the model you actually chose. What holds regardless is the shape of the growth:

Context lengthRelative KV cache size (modeled)
4k tokens1x baseline
8k tokens2x
32k tokens8x
128k tokens32x

Linear, in other words, and unbounded until you bound it. A 20GB budget with 4GB of headroom at 4k has no headroom at all at 32k.

The practical move is to cap context deliberately rather than discover the ceiling in production. Most runtimes let you set a maximum context at load time, and setting it to what you actually need converts an unpredictable crash into a predictable, bounded reservation. It is the difference between a system that fails at the worst moment and one that refuses politely.

What happens when it almost fits?#

It runs, slowly. Most runtimes will offload the layers that do not fit to system RAM and keep going, which is why a setup can look successful and feel unusable. Throughput falls sharply once any meaningful share of the model lives off the card, and the drop is not gradual.

This is worth understanding because it changes how you read a successful install. The model loaded and answered, so nothing reported an error, and the only symptom is that responses take many times longer than the benchmarks you read. People often conclude the hardware is simply slow and stop there.

Check where the layers actually went. Every serious runtime reports how many were placed on the GPU versus offloaded, and that number tells you immediately whether you are running on the card or beside it. If a meaningful share is off-card, dropping one precision level usually buys more speed than any other single change, because it moves the whole model back onto the GPU.

How do you size it in five steps?#

Pick the model, pick the precision, multiply for the weights, add the KV cache at the context length you actually use, then leave headroom. Headroom is not optional: the loading spike, the operating system, and anything else on the card all take a share before your first request arrives.

Five-step diagram: pick the model, pick the precision, compute weight memory, add the KV cache at real context length, then add headroomStep four is the one people skip, because the context length they test with is not the one they ship with.

The step people skip is the fourth one, and they skip it because the context length they test with is not the context length they ship with. Size against your real ceiling, not your demo.

What does quantization actually cost you?#

Quality, and not uniformly. Four-bit quantization roughly quarters the memory and does degrade output, but it shows up far more on reasoning and code generation than on summarisation or classification. Test your own task at both precisions before deciding, because the answer differs by workload more than any general rule admits.

Three cards showing bytes per parameter, maximum context length and peak concurrent requests as the inputs to a VRAM budgetEverything else is derived. Get these three wrong and no amount of benchmarking rescues the estimate.

It is worth saying plainly because most coverage presents low precision as a free win. It is not free. It is a trade that is usually worth making, which is a different claim. If your workload is classification or extraction, you will likely never notice. If it is multi-step reasoning inside an agent, you might notice a lot, and the failure will look like the model getting worse at its job rather than anything to do with memory.

A practical route: run your real prompts at INT8 and INT4, compare the outputs yourself, and only then decide how much memory you are buying. This is the same discipline as choosing between local and API in the first place, where the honest answer also depends on your specific numbers rather than a general rule.

What breaks the calculation?#

Five things, and concurrency is the one people miss. Each simultaneous request carries its own KV cache, so serving four users at once multiplies that part of the budget by four. Add the loading spike, other processes on the card, and the quality cost you accepted at low precision.

Checklist of five things that make a VRAM calculation wrong: context length, concurrency, loading spike, other processes, and quality lossEach one makes the real requirement larger than the number you calculated, which is the direction that hurts.

Concurrency deserves the emphasis because it converts a personal setup into a shared one. A card that comfortably serves you at your desk can fall over the moment three colleagues start using the same endpoint, and nothing about the model changed. If you plan to share it, size for peak concurrent requests rather than average ones.

There is a cheaper answer than a bigger card, and it is worth trying first. Serving stacks can queue requests rather than run them all at once, which trades a little latency for a fixed memory ceiling. Two requests served back to back need one KV cache; two served simultaneously need two. If your traffic is bursty rather than genuinely parallel, a queue is the difference between the card you have and the card you were about to buy.

Should you rent or buy?#

Rent first, almost always. A card is a capital purchase against a workload nobody has measured yet, and hourly rental answers the question for the price of an afternoon. Buy once you know the model, the precision, the context length and the concurrency, and the machine will stay busy.

Decision flowchart for renting versus buying a GPU for a self-hosted language model based on measured load and data residencyTwo questions settle it: whether the workload has been measured at all, and whether the data is allowed to leave the building.

The case for buying is real but narrower than enthusiasm suggests: steady, predictable load, data that cannot leave your building, or a rental bill that has clearly overtaken the hardware. The case against is that utilisation is usually far lower than expected, and an idle card depreciates just as fast as a busy one.

Data residency is the one argument that survives every spreadsheet. If the material genuinely cannot leave your premises, rental economics stop being the question and the only decision left is which tier. Everything else is arithmetic you can redo when the numbers change.

A reasonable sequence: rent the tier your calculation suggests, run a real week of traffic through it, and look at two numbers afterwards. Peak memory tells you whether the tier was right. Utilisation tells you whether owning it would have been idle capital. Most people who run that week end up renting a smaller tier than they were about to buy.

If you are still setting the stack up, the local runner comparison covers what to run the model with, and wiring it into n8n covers what to do with it once it answers. For a private document setup specifically, local RAG with Ollama is the worked version, and the VPS setup guide covers the host around it. If the bill is what brought you here, the token math and the cost levers are the two to read next.

Frequently asked questions

How much VRAM do I need for a 7B model?
Roughly 14GB for the weights at FP16, 7GB at INT8 and about 3.5GB at INT4, before the KV cache. Add the cache for the context length you actually run and leave headroom for the loading spike and the operating system. At long context the cache can rival the weights.
What is the KV cache and why does it matter for VRAM?
Every token in the context window stores a key and value entry so the model does not recompute past attention. That memory grows with conversation length, so it is the reason a model that loaded comfortably runs out of memory later. It is also per request, so concurrency multiplies it.
Does quantization hurt model quality?
Yes, and unevenly. Four-bit quantization roughly quarters memory and degrades output more on reasoning and code generation than on summarisation or classification. Whether that matters depends on your workload, so run your own prompts at two precisions and compare rather than trusting a general figure.
Should I buy a GPU or rent one?
Rent first in almost every case. Hourly rental answers the sizing question for the price of an afternoon, while a card is capital committed against a workload nobody has measured. Buying makes sense with steady predictable load, data that cannot leave your building, or a rental bill that has overtaken the hardware.
Why do VRAM guides disagree with each other?
Because most publish a model-to-card table rather than the calculation, and the tables age badly as models and hardware change. They also usually quote weights only, omitting the KV cache and any headroom, which makes the numbers look smaller than what you need in production.

Written by

Muhammad Qasim Hammad
Muhammad Qasim Hammad
AI agents & automationFounder · Cart Gaze LLCPMP-certified PM

Muhammad Qasim Hammad is an AI agent and automation expert and the founder of Cart Gaze LLC (cartgaze.com). He builds product for the love of it: when an idea lands, a working prototype is usually running within hours, built with the same AI agents and automations he sells. He puts his own output at roughly 20× what it was before agents, and the Agentic OS behind this site is the working proof, documented in public with the tools he actually ran and what they really cost.

AI & Automation Services

Want a pipeline like this running in your business?

I'm Qasim — I design and ship AI agents and n8n automations for solo operators and small teams. Tell me what's eating your team's week, and I'll scope a fix.

Related reading