KV Cache Explained: Why Long Context Costs More VRAM Than Your Weights

Your model loads. The GPU shows weights resident with room to spare. Short prompts come back in a few hundred milliseconds. Then someone pastes a 200-page contract into the box and the server dies with an out-of-memory error before the first output token appears. Nothing about the weights changed. They are the same bytes they were a minute ago.

What changed is a second allocation that almost nobody puts in the sizing spreadsheet: the KV cache. It is not a fixed cost. It grows linearly with every token in the sequence and linearly with every concurrent sequence, and past a certain context length it is bigger than the model it serves. This is the most common reason a deployment that "fits" does not actually run.

Why the cache exists at all

For each token, attention projects the hidden state into a query, a key, and a value. The query for the current token is compared against the keys of every prior token to produce attention weights, which then take a weighted sum over the values of every prior token.

The critical property: generating token N requires the keys and values of tokens 1 through N-1. Not the hidden states, not the logits, not the text. The K and V projections specifically, at every layer, for every prior position.

Consider generation without a cache. To emit token 1 you run a forward pass over the prompt. To emit token 2 you run a forward pass over the prompt plus token 1, recomputing K and V for every position you computed one step earlier. Producing N tokens means N passes over progressively longer sequences, so total projection work grows with the square of the output length. A 4,000-token response would redo the same arithmetic thousands of times.

But K and V for position i never change once computed. Position i does not attend forward, and nothing later modifies it. So compute them once and keep them. Each decode step then projects K and V for exactly one new token and appends.

This turns quadratic recomputation into linear work, and pays for it in memory proportional to everything the model has seen. That trade is what the OOM is telling you about.

The formula

kv_bytes = 2 x num_layers x num_kv_heads x head_dim x bytes_per_element x seq_len x batch_size

The 2 is keys and values: two tensors of identical shape. It is the term people drop most often, and dropping it halves every estimate.

num_layers is the transformer block count. Every layer keeps its own independent cache.

num_kv_heads is key/value heads per layer. Read it from the config, not from the attention head count. More on this below.

head_dim is head width, commonly 128. Often hidden_size divided by num_attention_heads, but not always, so read it explicitly.

bytes_per_element is cache precision, independent of weight precision: 2 for FP16 and BF16, 1 for FP8 and INT8. You can serve FP8 weights with an FP16 cache or the reverse. Separate knobs.

seq_len is prompt tokens plus tokens generated so far. batch_size is sequences resident at once.

The last two are the only terms you control at serve time, and the relationship is linear in both. Double the context, double the cache. Double the concurrency, double the cache. No economy of scale hides anywhere. That is exactly why context length and concurrency trade against each other: they multiply into the same fixed pool of VRAM.

Collapse the architectural terms into one constant and the rest is multiplication:

bytes_per_token = 2 x num_layers x num_kv_heads x head_dim x bytes_per_element

GQA and MQA: why long context became affordable

Classic multi-head attention gives every query head its own KV head. Sixty-four query heads, sixty-four KV heads. Multi-query attention goes to the other extreme: all query heads share a single KV head. Grouped-query attention sits between, partitioning query heads into groups that each share one KV head. The memory ratio is exactly num_kv_heads / num_attention_heads.

Take 80 layers, 64 query heads, head_dim 128, FP16:

• MHA, 64 KV heads: 2 x 80 x 64 x 128 x 2 = 2,621,440 bytes per token, or 2.5 MiB
• GQA, 8 KV heads: 2 x 80 x 8 x 128 x 2 = 327,680 bytes per token, or 320 KiB
• MQA, 1 KV head: 40 KiB per token

At 128k context, batch 1, that is 320 GiB for MHA against 40 GiB for GQA. The MHA figure fits on nothing most readers own. Near-universal GQA adoption is not a minor optimization; it is why 128k-plus context is servable outside a datacenter at all.

The warning: num_kv_heads is the term in the formula, not num_attention_heads. Substituting the attention head count inflates the example eightfold. If your hand calculation is wildly larger than what the engine allocates, this is almost always why. In Hugging Face configs the field is num_key_value_heads; when absent, the model is MHA and the two are equal.

A worked table

An illustrative configuration, deliberately not any released model: 80 layers, 8 KV heads, head_dim 128, GQA, batch size 1. That gives 320 KiB per token in FP16 and 160 KiB per token in FP8.

Nothing here is surprising once you accept linearity, but the magnitudes are. At 8k the cache is a rounding error next to the weights. At 128k it is a second model. At 900k in FP16 it exceeds what two 128 GB machines hold combined, before a single weight is loaded.

Community testing reports a 2x DGX Spark deployment, 128 GB per unit for 256 GB combined, running Qwen3.8-Flash-Next at 900k context with vision on SGLang with multi-token prediction and an SM121 kernel patch, stress tested at a 300k-token prefill. You do not need their config to know something firm: weights plus 900,000 tokens of cache plus activations plus overhead all had to fit under 256 GB. Rearranging gives the ceiling directly.

max_bytes_per_token = kv_budget_bytes / max_context_tokens

With 150 GiB available for cache, 900,000 tokens allows roughly 175 KiB per token. The illustrative FP16 geometry at 320 KiB is disqualified immediately; the FP8 variant at 160 KiB fits with margin. Long-context records are won by driving bytes-per-token down, not by having more memory.

Concurrency is the same budget, divided

total_kv = bytes_per_token x sum(seq_len across active sequences)

Suppose 96 GiB remains after weights and overhead, at the FP8 illustrative 160 KiB per token. That holds about 629,000 tokens total, however you slice them:

• 32k per session: 19 concurrent sessions
• 128k per session: 4 concurrent sessions
• 900k for one session: does not fit

This is the real capacity limit of a serving box, and it is usually not compute. Community testing on that 2x Spark deployment measured roughly 64 tok/s single stream and roughly 115 tok/s aggregate across 2 to 4 concurrent sessions. That shape is the signature of the regime: per-stream throughput falls as sessions are added while total throughput rises, because decode is memory-bandwidth bound and the streams share one bandwidth pool. Adding sessions extracts more aggregate work from the same hardware, right up to the point where the cache has no room for another sequence. Then the next request does not slow down, it is refused.

A second-order effect is worth internalizing: every decode step reads the whole cache for that sequence, not just the weights. Take an 18B-active MoE at FP8, GLM-5.3-Flash's stated active count at 320B total and roughly 328 GB of FP8 weights, on 4x DGX Spark at TP=4, which community testing puts at roughly 1092 GB/s aggregate bandwidth. Weight traffic per step is about 18 GB, a roofline near 60 tok/s with an empty cache. Add 128k of context at the illustrative FP8 geometry and you add 20 GiB, roughly 21.5 GB, read every step. Total traffic roughly 39.5 GB, roofline roughly 28 tok/s. These ceilings ignore everything else, but the direction holds: long context costs throughput continuously, not just capacity once.

PagedAttention: stop reserving what you never use

Naive implementations allocate one contiguous buffer per sequence, sized for max_model_len, because the cache must grow and reallocating mid-generation is painful. With max_model_len at 128k and a typical request using 4k, 96.9 percent of every reservation is never touched. Your server refuses the fifth request while nearly all its cache memory sits idle.

PagedAttention borrows the operating system's answer to exactly this. The cache is split into fixed-size blocks, often 16 tokens, with a per-sequence block table mapping logical positions to physical blocks. Blocks are allocated on demand and returned to a shared pool when a sequence finishes. The attention kernel gathers from scattered blocks instead of assuming contiguity.

Waste drops to at most one partial block per sequence. Sequences of wildly different lengths pack into one pool without fragmentation. And blocks become shareable: two sequences with identical leading tokens point at the same physical blocks, copying only where they diverge. This is why vLLM and engines using the same design reach several times the concurrency of naive implementations on identical hardware, with no change to the model.

Prefix caching: compute the shared part once

Block sharing generalizes. If two requests begin with the same tokens, their K and V for those positions are bit-identical, because attention at position i depends only on positions 1 through i. Compute the prefix once, reuse it.

The wins land in the workloads people actually run. A 2,000-token system prompt shared across every request is computed once. Multi-turn chat resends the whole conversation each turn, and every turn but the last is an unchanged prefix. Agent loops reuse a long tool-definition preamble across dozens of steps. RAG over a repeatedly-queried document caches the document and prefills only the question.

The effect is on time-to-first-token and prefill compute, not steady-state decode, and it does not reduce peak cache size for one long request. Order your prompt stable-part-first: shared instructions, then shared documents, then the varying turn. A per-request timestamp at the top of an otherwise identical system prompt invalidates the entire prefix and forfeits the whole saving.

Every lever, and what it costs

FP8 or INT8 KV quantization. Exactly halves the cache against FP16, usually a one-flag change. Cached K and V are fairly quantization-tolerant, but verify on your own evals rather than trusting that: degradation appears unevenly, and long-context retrieval is where it usually shows first. Kernel support varies by engine and GPU generation.

GQA or MQA. The largest factor by far, and not yours at serve time. It is baked into the checkpoint. Treat it as a model-selection criterion: compare num_key_value_heads before benchmark scores, because it decides whether the deployment is possible.

Lower max_model_len. Caps the worst case and immediately raises admitted concurrency. Costs nothing in quality for requests that fit; larger requests are rejected or truncated. Set it to what you serve, not what the model supports.

Paging. Recovers reservation waste at no quality cost. The price is engine complexity and a custom kernel, which is why you adopt an engine rather than build one.

Prefix sharing. Free when prefixes genuinely repeat, worthless when they do not, and it needs prompt discipline plus hash-table bookkeeping.

Sliding-window and local attention. A layer attending only to the last W tokens has its cache bounded at W regardless of sequence length. Models mixing local and global layers cut the cache substantially. The cost is real: information outside the window is reachable only indirectly, which hurts exact long-range recall. Architectural, not a serve-time switch.

Eviction and compression. Drop or merge low-attention entries, keep the first tokens plus a recent window, or compress KV to lower rank. These cut hard but are lossy by construction, and the loss is unpredictable: the evicted token is the one the next question asks about. Gate behind task-specific evals.

The prefill connection

A 300k-token prefill must materialize 300k tokens of KV before the first output token exists. At the illustrative FP8 geometry that is roughly 45.8 GiB written during prefill, and attention itself scales worse than linearly in prompt length. This is why TTFT climbs steeply with prompt size while decode speed stays roughly flat: prefill is compute-bound and bursty, decode is bandwidth-bound and steady.

It is also why chunked prefill exists. Processing a 300k prompt in one operation monopolizes the GPU and stalls every other session's decode. Splitting prefill into chunks of a few thousand tokens and interleaving them with decode steps keeps other sessions producing. Total prefill time is unchanged or slightly worse; tail latency across concurrent users improves a lot.

Practical checklist

1. Read num_hidden_layers, num_key_value_heads and head_dim from the config. Compute bytes_per_token and write it down.
2. Measure free VRAM after weights, activations and runtime overhead. That, not total VRAM, is your cache budget.
3. Divide. The resulting token count is your entire capacity, split any way you like between context and concurrency.
4. Set max_model_len to what you actually serve. Fastest concurrency win available.
5. Enable FP8 or INT8 KV quantization and re-run evals, especially long-context retrieval. Expect roughly double the capacity.
6. Use an engine with paging and prefix caching. Order prompts stable-part-first so prefixes actually hit.
7. If TTFT is the complaint, enable chunked prefill before touching anything else.
8. When an OOM fires at long context but not short, stop looking at the weights. Compute the cache at the failing sequence length first.

The weights are the number on the model card. The cache is the number that decides whether you ship.