Will It Fit? VRAM Math for Open-Weight Models Across Every Quantization Format

A model gets announced. You read "320B total, 18B active", halve it for FP16, get 640 GB, decide it is hopeless, close the tab. Or you read "70B", halve it, get 35 GB, look at your 48 GB card and start the download. One of those two people is about to be surprised.

Dividing parameter count by two is wrong in three directions at once. It assumes a quantized format stores exactly bits divided by eight bytes per parameter, which none of them do. It ignores the KV cache, which at long context routinely costs more than the weights. And for mixture-of-experts models it starts from the wrong parameter count entirely. The result is a floor. What you need is the floor plus three other things, and this is how to add them up for any model, including ones that do not exist yet.

The four components of the budget

Everything resident on the accelerator at steady state falls into one of four buckets:

Weights are the number everybody quotes. KV cache is the number that decides whether the deployment works. The other two are small individually and collectively large enough that planning to 100 percent of nominal VRAM fails.

One unit hazard first. Model files are sized in decimal GB (10 to the 9th bytes); allocators and nvidia-smi report binary GiB (2 to the 30th). The gap is 7.4 percent, about 9 GB on a 128 GB box — enough on its own to turn a fit into a miss.

1. Model weights

weights_bytes = params x bytes_per_param

Parameters in billions times bytes per parameter gives gigabytes directly, which makes this the easiest arithmetic in the exercise.

The subtlety is that bytes_per_param is never the nominal bits divided by eight. Low-precision formats store block scale factors alongside the elements, and no published checkpoint quantizes every tensor — embeddings, the output projection, normalization parameters and MoE router weights are routinely left at higher precision because they are disproportionately sensitive.

Sanity check against a known quantity: GLM-5.3-Flash is 320B parameters, FP8 weights roughly 328 GB. That is 1.025 bytes per parameter, right in the band above; the 2.5 percent over nominal is the un-quantized remainder plus alignment padding.

The rule that follows: when a published file size exists, use it. Divide repository size by parameter count for the exact ratio of that build. Use the table only when no file exists yet.

2. KV cache, the component that surprises people

Attention caches a key and a value vector for every token in every layer, so generating token N does not recompute attention over tokens 1 through N-1. It is the biggest single reason a model that loaded fine crashes forty minutes into a session.

kv_bytes_per_token = 2 x layers x kv_heads x head_dim x bytes_per_element
kv_bytes_total     = kv_bytes_per_token x context_length x batch_size

The leading 2 is keys and values — two tensors of identical shape per layer. bytes_per_element is 2 for a FP16 or BF16 cache, 1 for FP8, 0.5 for 4-bit. Note what is absent: hidden size, feed-forward width, expert count, total parameters. KV cost is a function of attention geometry alone, which is why models of wildly different size can have near-identical cache footprints.

The term that matters most is kv_heads, and it is the one people substitute wrongly. In classic multi-head attention it equals the number of attention heads. Grouped-query attention (GQA) shares one key/value pair across a group of query heads; multi-query attention (MQA) collapses to a single pair per layer. Read num_key_value_heads from the config, not num_attention_heads — the ratio between them is your cache divisor.

Take an 80-layer model with 64 attention heads and head_dim 128, BF16 cache:

• MHA, kv_heads = 64: 2 x 80 x 64 x 128 x 2 = 2,621,440 bytes per token, or 2.62 MB.
• GQA 8:1, kv_heads = 8: 2 x 80 x 8 x 128 x 2 = 327,680 bytes per token, or 0.33 MB.

An eightfold cut for a modest quality cost. At 128k context that is the difference between 344 GB of cache and 43 GB. GQA is not a minor optimization; it is the reason long context is offered at all. Any model advertising 200k or more without GQA or MQA expects you never to use it.

Now the point that matters. Same model, weights at 4-bit and 0.55 bytes per parameter, so 70B parameters is 38.5 GB. With the GQA figure of 327,680 bytes per token:

At roughly 115,000 tokens the cache costs more than the model. Past that point every additional GB of hardware buys context, not capability. And that is batch 1 — eight concurrent 32k sessions need 8 x 10.7 = 86 GB of cache, more than twice the weights, before any session is long.

Two mitigations change the arithmetic directly. KV quantization to FP8 halves bytes_per_element, and so halves the table above. Paged attention (the vLLM and SGLang allocator model) does not reduce the worst case, but it stops you reserving max context for every sequence regardless of actual length.

3. Activations and workspace

Transient buffers for the forward pass: layer inputs and outputs, attention scores, feed-forward intermediates, and the logits tensor.

activation_bytes ~= batch_size x chunk_size x hidden_size x bytes x k

where k is a small constant, typically 10 to 20 depending on how aggressively the engine fuses kernels. At batch 1, a 2048-token chunk, hidden size 8192 and BF16, one buffer is 34 MB and a dozen live buffers is around 400 MB. Small.

Two things make it not small. Prefill without chunking processes the whole prompt at once, so chunk_size becomes the full prompt length — a 300k-token prefill in one shot is a 150-fold multiplier. Chunked prefill is what keeps this bucket bounded, and it is the first setting to check when a long prompt OOMs on a model that generates fine. And the logits tensor is batch x chunk x vocab x 4 bytes; with a 150,000-token vocabulary at chunk 2048 that is 1.2 GB alone, which is why engines cap how many positions they compute logits for.

Budget 1 to 3 GB at batch 1 with chunked prefill on, and scale with batch.

4. Framework and fragmentation overhead

The CUDA context alone is 300 to 600 MB per device per process before you allocate anything. Add cuBLAS and cuDNN workspaces, NCCL buffers that grow with tensor-parallel degree, CUDA graph pools capturing the decode path, and allocator slack from blocks that cannot be coalesced.

None of it is precisely predictable, which is the point. Budget 5 to 10 percent of nominal device memory as headroom and do not plan to the last gigabyte. On unified memory the reservation is larger, because the OS, display stack and page cache draw on the same pool: a 128 GB unified box typically offers a serving process somewhere near 110 to 120 GB.

The MoE correction

This is where the naive calculation goes from slightly wrong to catastrophically wrong.

A mixture-of-experts model routes each token to a small subset of its expert feed-forward blocks. GLM-5.3-Flash is 320B total parameters with 18B active per token, and people read the second number and size hardware against it. But every expert must be resident: the router picks per token, so any expert may be needed for the next one and you cannot know which in advance. An 18B-active model needs memory for all 320B parameters.

Active parameter count predicts speed. Total parameter count predicts memory.

The speed half is arithmetic too. Decode is memory-bandwidth bound, so the throughput ceiling is roughly bandwidth divided by bytes read per token, and bytes read per token is the active parameters at their stored precision:

tokens_per_second_ceiling ~= memory_bandwidth / (active_params x bytes_per_param)

At FP8, 18B active is 18 GB per token. Four DGX Spark units at TP=4 give roughly 1092 GB/s aggregate, so the ceiling is about 60 tokens per second; a 512 GB M5 Ultra at roughly 1200 GB/s gives about 66. Real numbers land below these, since this ignores attention, communication and kernel overhead. But the shape explains why a 320B MoE feels like an 18B model to talk to while occupying the memory of a 320B one.

Expert count does not affect the KV cache at all, incidentally. Cache size depends on attention geometry only.

Worked example 1: a 320B/18B model at FP8

Weights: 328 GB, measured, not estimated. Add a modest 32k context with GQA-class geometry (call it 10 GB of cache), 3 GB of activations and framework overhead, and you need roughly 350 GB usable — so for any margin, a platform in the 400 GB class or above.

Now change one variable. At 4-bit, 320B x 0.55 is 176 GB, and two 128 GB units become viable: 88 GB per unit under TP=2, leaving roughly 30 GB each for cache and overhead. A four-unit deployment becomes a two-unit one, purely on format.

The catch is real: GLM-5.3-Flash has no NVFP4 build as of writing. An FP8 checkpoint existing does not imply a 4-bit one will — somebody has to run the calibration and publish it. Size against the formats that exist today.

Worked example 2: a 135 GB model against a 128 GB box

Qwen3.8-Flash in NVFP4 is roughly 135 GB of weights. A DGX Spark is a 128 GB unit. The naive read is that you miss by 7 GB and some flag closes it. Work the actual budget:

You are not 7 GB short. You are about 34 GB short, and no flag closes that. Two units is the answer, and it is the documented one: community testing reports a 2x DGX Spark deployment reaching 900k context with vision under SGLang with MTP, stress tested at 300k token prefill.

But note what two units actually buys, because 2 x 128 GB is not 256 GB of usable model space. Under tensor parallelism the weight matrices shard cleanly, so each rank holds roughly 67.5 GB. The KV cache shards too, attention heads splitting across ranks. What does not shard:

• Per-device fixed costs are paid once per rank, not once per cluster. CUDA context, allocator pools and engine buffers are duplicated on every unit, and NCCL buffers grow with TP degree on top.
• Activations are replicated at layer boundaries. All-reduce operates on full-width tensors, so every rank materializes them.
• KV sharding has a floor. When TP degree exceeds kv_heads, the cache stops sharding and starts replicating — 4 KV heads at TP=8 duplicates the cache across pairs of ranks. Check num_key_value_heads against your intended TP degree before assuming linear scaling.

Practically, 2 x 128 GB behaves like about 230 GB, not 256, and you pay an interconnect tax on latency for every layer of every token.

Worked example 3: where long context takes you

Run the same 80-layer, 8-KV-head, head_dim 128 model to the extremes, weights at 4-bit (38.5 GB), cache at both BF16 and FP8:

At 8k the cache is a rounding error. At 900k it is four times the model, and the model is irrelevant to the sizing decision.

Now invert the reported result. Two Spark units is 256 GB nominal, call it 230 GB usable. Subtract roughly 135 GB of weights and roughly 15 GB of activation, vision-encoder and framework overhead, and something near 80 GB is left for cache. Divide by 900,000 tokens:

80 GB / 900,000 tokens ~= 89 KB per token

That is the ceiling such a deployment lives under, and it is demanding — roughly a quarter of the 327 KB per token our BF16 example needs. Getting there means some combination of few KV heads, few layers relative to parameter count, a quantized cache, and MTP amortizing decode. The lesson holds whichever lever did the work: at extreme context you are not buying memory for the model, you are buying it for the cache, and attention geometry decides how far the money goes.

Quick-reference fit table

Parameter ranges rather than named models, so this does not go stale. Assumes batch 1, context up to 32k, GQA-class attention, and roughly 75 percent of nominal memory budgeted for weights. At 4-bit use 0.55 bytes per parameter, at 8-bit 1.05.

For MoE models, read the total parameter count against this table, never the active one. Beyond 64k context, drop the weight budget to 60 percent — or better, run the cache formula properly.

When it does not fit

In order of preference. The early rungs cost you little; the later ones cost you a lot.

1. Take a smaller quantization. FP8 to 4-bit nearly halves the weights for a usually modest quality cost. The cheapest 45 percent you will ever find — if a build exists.
2. Cut max context. Most deployments configure 128k and serve prompts averaging under 8k. The engine reserves against the configured maximum, so lowering it frees real memory at zero quality cost for the traffic you actually have.
3. Quantize the KV cache. FP8 halves that bucket, and on a long-context deployment that frees more memory than any other single change. Quantize keys more conservatively than values.
4. Tensor parallel across devices. Real capacity gain, at the cost of an interconnect on the critical path of every layer. Keep TP degree at or below num_key_value_heads or the cache stops sharding.
5. Pipeline parallel. Splits by layer rather than within layers, so it needs far less interconnect bandwidth. Good for slow links, bad for single-stream latency: only one stage is busy at a time unless enough concurrent requests fill the pipeline.
6. CPU or RAM offload. Be blunt: PCIe offers tens of GB/s against the hundreds or thousands of GB/s of device memory, and any weight in host RAM crosses that link on every token that needs it. Expect single-digit tokens per second where you had tens, and worse than the raw ratio suggests once transfer and compute fail to overlap. It makes a model technically run, not usable. The one defensible case is MoE expert offload with strongly skewed routing and hot experts pinned resident — and even then, measure first.
7. Accept a smaller model. Often right, rarely considered first. A model that fits with room for your real context length, served at full speed, beats a larger one thrashing against an offload boundary on every metric a user perceives.

The checklist

Against any newly announced model:

1. Find total parameter count, not active. For MoE these differ by an order of magnitude and memory tracks the total.
2. Find the published file size for the format you intend to run. That number is ground truth; divide by parameter count for real bytes per parameter. If no such build exists, confirm someone is shipping one before planning around it.
3. Read num_hidden_layers, num_key_value_heads and head_dim from the config. Not num_attention_heads.
4. Compute 2 x layers x kv_heads x head_dim x bytes_per_element for one token, then multiply by the context you will genuinely configure and the concurrency you genuinely need.
5. Add 1 to 3 GB for activations at batch 1 with chunked prefill on, more if you batch.
6. Add 5 to 10 percent of nominal device memory for framework overhead, more on unified memory.
7. Compare against usable memory, not the number on the box, and confirm you are comparing GB to GB rather than GB to GiB.
8. If it misses, walk the ladder above from the top. If you reach its step 6, go back and reconsider its step 7.

The arithmetic takes five minutes, and it is the same five minutes for every model that will ever be released. Formats change, architectures change, the numbers get larger. The four buckets do not.