Your agent re-sends the same long prefix on every step and pays to recompute it. Here is the KV mechanism, the ordering rule it forces, and the break-even math.
An agent run looks like this from the API's side. Twelve calls, each carrying the same system prompt, the same tool schemas, the same reference material, and a transcript grown by one turn. If the fixed portion is 20,000 tokens and each step adds a few hundred, you sent roughly 240,000 input tokens across the run, about 220,000 of them byte-identical repeats of what you sent seconds earlier.
The model has no memory of any of that. Without caching, every call prefills the whole prompt from scratch: attention over all 20,000 fixed tokens, twelve times, producing identical tensors every time. Eleven of those computations are discarded work, paid for in dollars and in time-to-first-token.
Removing that repeated work is the largest cost and latency lever most teams have. It needs no model change, no distillation, no quantisation, no cutting of the prompt — only one mechanism understood and a prompt arranged to suit it.
What is actually being cached
The name misleads: the cache does not hold responses. A cached request still runs the model, still samples, still returns fresh output.
What is cached is the KV cache — the key and value tensors the transformer computes for every token during prefill, at every layer. Each position produces a key and a value vector per attention head, and those are what later tokens attend against. They are the expensive intermediate product of prefill, and why a long prompt costs more than a short one.
What makes them reusable is causality. Attention is masked so position i never sees position i+1, so a position's tensors are a deterministic function of that token and everything before it, and of nothing after. Run the same 20,000 tokens through the same model twice and you get the same tensors twice. Compute them once, keep them, and a later request beginning with those tokens loads them and starts prefill at position 20,001.
This is the same KV cache that already governs serving memory — the structure that lets each generated token attend against stored keys and values instead of recomputing the sequence, and whose per-sequence size makes long contexts expensive to serve. Ordinary use reuses it within one request, across decode steps. Prefix caching reuses it across requests. Same data, different lifetime.
The prefix property
Because every position depends on everything before it, changing a token at position j changes the keys and values at every position from j onward. Nothing after an edit survives it.
The cache is therefore valid only for an exact prefix match from position zero. Not "mostly the same", not "the same paragraph somewhere in the middle". Exact, contiguous, from the first token; the reusable portion ends at the first byte that differs.
Everything about prompt architecture follows: static first, dynamic last. From the top:
1. System instructions and role definition
2. Tool and function schemas
3. Long static reference material: policies, API specs, code, few-shot examples
4. Session-stable retrieved context, if any
5. Conversation history, appended in order
6. The new user turn
And the mistake worth calling out sharply, because it is everywhere: put a timestamp, a session id, a request id, or a trace token in the first line of your system prompt and your hit rate is zero, permanently. Every request differs within the first hundred tokens, so nothing after them is reusable. The prompt is ninety-nine per cent identical and zero per cent cacheable.
Nothing fails when you do this. No error, no warning, no degraded output — the system just costs what it always cost, and the team concludes caching did not help much. If the model needs the clock, put it in the user turn or behind a tool.
Two places caching happens
Provider-side caching on hosted APIs
You mark a prefix explicitly or the provider detects a repeated one; it stores the KV state keyed to that prefix and serves later requests beginning with it without re-prefilling, billing the cached portion differently. Parameters vary by provider and change often, so learn them as categories and read current documentation for values:
- Explicit or automatic. Some APIs need you to mark a cache breakpoint; others detect repeated prefixes. This decides whether cacheability is declared or reverse-engineered from usage numbers.
- Entry lifetime. Unused entries expire. Whether a hit extends the lifetime, and whether longer-lived tiers exist at a different price, decides how a slow conversation behaves.
- Minimum cacheable length. Below some length nothing is cached at all, however often it repeats.
- Write cost. Populating the cache may be billed at a premium over ordinary input — what makes a single-use prefix worse than no caching.
- Cached input price relative to fresh input. The discount is the whole economic point, and the number to look up rather than assume.
- What else is in the key. Model version and account scope are usually part of cache identity, and matches may round down to a block boundary.
Do not hard-code any of these. Read the provider's current pages, then verify against your own token counts.
Self-hosted prefix caching
In your own serving stack the same reuse is an engine feature you control. Paged KV stores the cache in fixed-size blocks addressed through a block table rather than one contiguous per-sequence buffer, so sequences sharing a prefix point at the same physical blocks instead of each holding a copy. Radix-tree indexing keeps computed prefixes in a tree, so an arriving request finds its longest cached prefix cheaply, with least-recently-used pruning for eviction. The tradeoff is explicitly yours: GPU memory holding KV blocks against compute recomputing them.
The second-order effect is often larger than the first. A shared prefix is stored once rather than per session, so two hundred sessions over a common 20,000-token prefix store it once and the memory lost to duplicates goes to more concurrent sequences. That raises effective concurrency and batch size, and therefore throughput, independently of the prefill you skipped.
The caveat: hit rate follows memory pressure, so the cache degrades when you are busiest. Measure at peak.
The arithmetic
Split each request's input into a cached prefix of P tokens and fresh tokens F, total N = P + F. Let c be the price of one fresh input token, and r the price of a cached input token divided by c — a fraction less than one, whose value you look up.
cost without caching = N * c
cost with a hit = (P * r + F) * c
savings fraction = (P / N) * (1 - r)
Two terms, one of them yours. The provider sets 1 - r; you set P / N, and the next section is entirely about moving it toward one.
Now the write. Let w be the price of writing one token into the cache divided by c; it may exceed one. Over k requests sharing one prefix — one write, then k-1 hits:
with caching = P * (w + (k - 1) * r) + k * F
without caching = k * (P + F)
break-even = k must be at least (w - r) / (1 - r)
Fill in your provider's numbers and you get a reuse threshold. If writes carry no premium, w = 1 and one reuse already pays. If they do, you need enough repeats to amortise it, and a prefix used exactly once costs more than not caching at all.
The shapes matter. An agent with S steps over a fixed prefix prefills roughly S * P uncached and P once cached — two orders of magnitude with S in the double digits. Chat over a fixed document puts P / N above 0.99, so savings approach the full 1 - r. One-shot requests with short unique prompts have P near zero, so savings are near zero, and with a write premium you are behind.
Latency lands somewhere specific. Prefill is the part of inference that scales with prompt length, and prefill determines time-to-first-token; decode speed after that is governed by generation length and memory bandwidth, not prompt length. A hit therefore shows up almost entirely in TTFT, in rough proportion to the prefill skipped:
TTFT on a hit is approximately TTFT on a miss * (F / N), plus lookup overhead
That is a measurement warning too: benchmark end-to-end latency on a long generation and decode dominates, hiding the gain.
Designing a prompt for cacheability
- Fix the order and freeze it. The ordering above determines how much of your prompt is reusable, so treat prompt layout as a contract. Keep everything volatile — timestamps, request ids, random identifiers, experiment flags — below the conversation history.
- Serialise tool schemas once, deterministically. A schema regenerated from a dictionary per call can reorder keys, change whitespace, or format numbers differently after a library upgrade. Sorted keys, fixed separators, serialised at process start and reused as a string. A silent reordering of two properties is a full cache miss with no other symptom.
- Append history, never edit it. Rewriting an earlier turn invalidates from that turn onward — exactly what compaction does. Compaction is still worth doing; schedule it deliberately and expect one full prefill afterwards rather than meeting the cost by accident.
- Put retrieved documents after the stable prefix. A retrieval pipeline injects different chunks per query, so that portion cannot cache. What is avoidable is letting it poison everything above it: many RAG templates put retrieved context at the very top for salience, placing variable content at position zero and forfeiting the entire prefix.
- Batch variable content at the tail, since each interleaved variable region truncates the prefix at its own position. Where the API has explicit breakpoints, put them on stable boundaries — end of tool schemas, end of the document block, a slow-moving point in history.
- Prefer fewer, longer-lived prefixes. Every distinct system prompt is its own entry, so four variants in an experiment split traffic four ways and may push you under break-even.
Where it fails or misleads
Nondeterministic serialisation. Dictionary iteration order, float formatting, a tool list built from a set, an encoder that changes separator defaults between versions. One differing byte and the prefix diverges. No error is raised; the request just costs full price.
Expiry between turns. A human thinking for minutes between messages can outlast the entry lifetime: turn one writes, turn two misses and writes again, so you pay write pricing repeatedly and collect no hits.
High variability. Per-user system prompts, per-request persona injection, dynamic tool subsets. Each variant is its own prefix with a small reuse count, and the write premium may exceed savings on most.
Minimum length and eviction. Short prefixes may be ineligible entirely; self-hosted, hit rate falls as memory pressure rises.
And the general one: measure hit rate, do not assume it. Providers return cached and fresh input token counts in the response usage; log them per request.
Security and privacy
A cache entry is state derived from your prompts, held beyond the request that created it. That makes it a small data store: what is in it, how long it lives, who can address it.
Tenant boundaries are the real risk. If lookup is keyed on prompt content, any request reproducing a prefix can in principle hit an entry another request created. Hosted providers scope entries to your account or organisation — confirm that in their documentation rather than assuming it. On your own stack, put the tenant identifier in the cache scope so lookups cannot cross boundaries, rather than relying on content being unguessable.
The simpler discipline sidesteps the question: shared prefixes should contain only shared content — instructions, schemas, public documents — with anything tenant-specific after it, scoped to that tenant. If no tenant data ever enters a shared prefix, cross-tenant addressing has nothing to leak.
Cache-hit timing is observable. A hit returns its first token measurably sooner than a miss, so someone who can submit prompts and time responses can in principle learn whether a prefix has been seen recently. The impact is modest, but it argues against treating a system prompt as a secret.
What to monitor
Cache hit rate, as cached input tokens over total input tokens, and as the fraction of requests with any hit: the first says how much input is reused, the second how many requests benefit.
TTFT split by hit and miss, as two distributions rather than one average. If they converge, you are not getting the hits you think you are.
Cost per request decomposed into cached input, fresh input, cache writes, and output. Write volume that does not fall relative to hits means prefixes that never get reused — the break-even formula failing in production.
Hash the prompt template and log it per request, so a hit-rate cliff after a deploy points straight at the commit.
Checklist
- Prompt ordered static to dynamic, nothing volatile above the conversation history.
- Tool schemas serialised deterministically, once, and reused as a string.
- Retrieved documents after the stable prefix, and history appended rather than rewritten.
- Cache scope unable to cross tenant boundaries, no tenant data in a shared prefix.
- Provider's current lifetime, minimum length, write cost, and cached price read from live documentation.
- Break-even reuse count computed for your own prefix, price ratio, and traffic.
- Hit rate, TTFT by hit and miss, and cost split logged against a prompt hash.
The mechanism is simple and the constraint is one rule. Nearly all the value is in respecting the prefix property everywhere in your prompt assembly code, then checking the token counts to confirm you did.