How to Optimize KV Cache

One user holding a 128K context conversation on a 70B model occupies 40 GiB of KV cache. That is half an 80 GB GPU, for one person, before anyone else connects. Run the math for your own model and context length and you will probably find your real concurrency ceiling is somewhere in the single digits, which is not the number on your capacity plan.

KV cache is what actually limits an LLM deployment. It sets how many requests you can hold in flight, how long a context you can offer, how often the scheduler preempts someone mid generation, and what a million served tokens costs you. Most teams are running at a fraction of their hardware’s real capacity, not because they bought the wrong GPUs, but because the cache is sized for a context length they never serve, reuse is enabled but never hitting, and everything sits in HBM at full precision.

The frustrating part is that KV cache is also the part of the stack where the most leverage hides. Most teams are leaving somewhere between two and ten times their effective capacity on the table, not because they picked the wrong GPU, but because the cache is sized wrong, reuse is turned off or silently not hitting, precision is higher than it needs to be, and nothing is tiering cold blocks out of precious HBM.

This guide walks through all of it: the math first, so you can compute your own numbers instead of trusting a blog post's, then each optimization layer in the order you should apply it, with the real flags and the metrics that tell you whether it worked. The examples lean on vLLM and SGLang because that is what most self hosted serving runs on, and every flag here comes from their current docs.

First, the Math

During prefill the model processes your whole prompt and writes a key and value vector for every token at every layer. During decode, each new token attends over all of that, so the cache only grows. The size is fully determined by the model's architecture:

KV bytes = 2 × layers × kv_heads × head_dim × tokens × bytes_per_element

The leading 2 is for keys and values. Everything else is a property of the model except tokens, which is your workload, and bytes_per_element, which is the one you can change at serving time.

Run that for two models people actually deploy. Llama 3.1 8B has 32 layers, 8 KV heads, head dim 128. Llama 3.3 70B has 80 layers, 8 KV heads, head dim 128.

That is per request. One user at 128K context on a 70B model occupies 40 GiB of KV cache in FP16, which is half an 80 GB GPU for a single conversation.

Now turn it around into the number you actually care about, which is concurrency. Take Llama 3.1 8B on one 80 GB GPU at gpu_memory_utilization=0.9. Weights take about 16 GB in BF16, and leave roughly 3 GB for activations, CUDA graphs, and overhead. That leaves about 49 GiB for KV:

And Llama 3.3 70B on four 80 GB GPUs with tensor parallelism, where KV heads shard across ranks so the capacity adds up:

Two things jump out of those tables. Your concurrency at long context is embarrassingly low, often single digits. And a single precision change doubles it.

One more number worth internalizing, because it explains why grouped query attention is in every modern model: if that same 70B used full multi head attention with 64 KV heads instead of 8, its KV cache at 128K context would be 320 GiB rather than 40 GiB. The architecture already did an eight times optimization for you before you touched a flag.

The Optimization Stack

It helps to think of KV cache optimization as layers, applied bottom up. Each one multiplies against the others, and the cheap ones are at the bottom.

The ordering matters. There is no point building a multi tier offload setup if prefix caching is not hitting, and no point tuning block sizes if your context length is set to 1M for a workload that never exceeds 8K.

Layer 0: Architecture, the Decision You Already Made

You cannot change this at serving time, but you should know what you picked, because it sets the ceiling for everything else.

Grouped query attention (GQA) shares one KV head across a group of query heads. Llama 3.x models use 8 KV heads against 32 or 64 query heads, which is where the eight times saving in the math above comes from. This is the default in essentially every modern open model.

Multi query attention (MQA) takes it to the extreme with a single KV head. Maximum saving, measurable quality cost, which is why GQA won as the default compromise.

Multi head latent attention (MLA), used by the DeepSeek family, compresses keys and values into a shared low rank latent vector and reconstructs them during attention. It gives a significantly smaller cache than GQA at comparable quality, and it is the main reason DeepSeek style models are attractive for long context serving specifically.

Sliding window attention (SWA) caps how far back each layer attends, so the per layer cache stops growing once the window fills. Models like the Mistral family and several recent hybrids use it, often interleaving sliding window layers with full attention layers. The tradeoff is real: you lose exact long range attention in those layers.

That last one creates a wrinkle worth knowing about if you serve hybrid models. A sliding window layer and a full attention layer share the same token prefix but not the same reusable boundary, since full attention KV stays reusable across the whole matched prefix while SWA KV only covers the trailing window. SGLang's unified radix cache work, published in August 2026, addresses exactly this by keeping one token keyed radix tree where full attention and SWA are separate components, each with its own reuse rules, and a candidate boundary is only accepted if every component accepts it. Before that, each model combination needed its own cache implementation.

Layer 1: Allocation and Sizing

This is the unglamorous layer where most of the easy wins live.

Paged allocation is the substrate. PagedAttention, which vLLM introduced and everyone now implements some version of, stores the cache in fixed size blocks with a block table, the same trick an operating system uses for virtual memory. Without it, you pre allocate contiguous space for each sequence's worst case length and waste most of it. You do not turn this on, it is simply how modern engines work, but it is worth understanding because it is why block size and fragmentation are things you can reason about at all.

Set max_model_len to what you actually serve. This is the single most common misconfiguration I see. Advertising a 1M context window on a workload where p99 prompt length is 12K means the engine sizes its scheduling and memory assumptions around a request that never arrives. Measure your real distribution and set it with headroom, not aspiration.

Push gpu_memory_utilization up carefully. vLLM pre allocates its KV cache from this fraction of GPU memory. The default is conservative. Raising it gives you directly proportional cache capacity, and the failure mode is an out of memory error at startup rather than a silent production problem, which makes it safe to tune empirically. Go up in steps, watch for OOM, back off one step.

Watch for preemption, and know what it means. When KV space runs out mid flight, vLLM preempts requests and recomputes them later, logging a warning like this:

WARNING Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because
there is not enough KV cache space. This can affect the end-to-end performance.
Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache
memory. total_cumulative_preemption_cnt=1

In vLLM V1 the default mode is RECOMPUTE rather than swapping, because recomputation has lower overhead in that architecture. Frequent preemption is the clearest signal you are cache starved, and the official remedies are exactly the levers in this section: raise gpu_memory_utilization, lower max_num_seqs or max_num_batched_tokens, or raise tensor_parallel_size so weights shard further and leave more room per GPU. Raising pipeline_parallel_size also distributes layers and indirectly frees memory, at a latency cost.

Tune max_num_batched_tokens against the latency you care about. Chunked prefill is enabled by default in vLLM V1 wherever possible, and it batches large prefills in pieces alongside decodes, prioritizing decode requests first. The tuning guidance from the docs is direct: smaller values such as 2048 give better inter token latency because fewer prefills interrupt decodes, higher values give better TTFT because more prefill tokens process per batch, and for throughput they recommend going above 8192, especially for smaller models on large GPUs.

# latency sensitive chat serving
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-batched-tokens 2048
 
# throughput oriented batch serving
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-batched-tokens 16384

One operational nicety: vLLM logs the exact --kv-cache-memory value that reproduces its current allocation, and passing that back on the next boot skips the memory profiling pass and speeds startup. Use it with care, since it pins the cache to exactly that size rather than measuring, so a conservative value caps your concurrency and an optimistic one fails at allocation. It is also only valid on the same GPU with the same free memory, so drop it if a boot OOMs after a hardware or co tenant change.

Layer 2: Reuse, Where the Big Wins Are

Everything so far was about fitting more cache in memory. This layer is about not computing it in the first place.

Automatic prefix caching (APC) hashes prompt blocks and reuses the KV cache of any matching prefix, so a new request skips the prefill compute for the shared part. In vLLM you enable it with enable_prefix_caching=True. The two workloads the docs call out as ideal are exactly the two most common production shapes: repeated queries against the same long document, where the document is processed once instead of every time, and multi round conversation, where the chat history is not reprocessed every turn.

Be precise about the limits, because this is where expectations go wrong. APC only accelerates prefill. It does nothing for decode. So if your workload spends most of its time generating long answers, or if requests genuinely share no prefix, the gain approaches zero. The docs are explicit that APC generally does not hurt performance, it just may not help.

Radix tree caching is SGLang's approach, where prefixes live in a tree rather than a flat hash map. The structural advantage shows up with branching: an agent loop or a sampled set of candidate continuations shares a trunk and diverges, which a tree represents naturally.

The part everyone discovers late: a cache hit requires landing on the instance that holds the cache. This is not a detail. In a multi replica deployment with round robin load balancing, a conversation's second turn has a one in N chance of hitting the replica that cached its first turn. You can have prefix caching enabled everywhere and still see a single digit hit rate. Fixing it means prefix aware or session aware routing, which is precisely what the Kubernetes Gateway API Inference Extension's endpoint picker and llm-d style schedulers exist to do, and what SGLang's HiCache docs mean when they note that L1 and L2 tiers are private to one instance so cross instance reuse requires a shared L3 backend.

How much does this matter? The vLLM and Mooncake Store integration published in May 2026 measured a real Codex agent trace workload and reported cache hit rate going from 1.7% to 92.2% once a shared cache pool replaced instance local caching. That is the whole ballgame for agentic workloads, and it is worth sitting with: the same engine, the same model, the same prefix caching feature, and a 54 times difference in hit rate purely from where the cache lived and how requests were routed.

Layer 3: Precision

FP8 KV cache is, for most deployments on Hopper or newer hardware, the highest return single flag you can set. It halves the cache footprint, which doubles either your concurrency or your context length, and the quality cost on most workloads is small.

In vLLM:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92

The kv_cache_dtype options are auto for the model's native dtype, fp8_e4m3 which works on CUDA 11.8 and later plus ROCm, and fp8_e5m2 on CUDA 11.8 and later. There are two quantization schemes: per tensor, with one scale for each of Q, K, and V, and per attention head, with a scale per head. Per head is more accurate but currently requires the Flash Attention backend and calibration through llm-compressor.

Two details that materially affect quality:

Calibrate your scales. With plain kv_cache_dtype="fp8" and no calibration, all scales are set to 1.0. The documented recommendation is to estimate scales from a calibration dataset using llm-compressor. Uncalibrated FP8 is the version of this feature that gives people a bad impression of it.

Skip the sensitive layers. Some attention layer types, sliding window layers in particular, are more sensitive to KV quantization than others. vLLM exposes a flag for exactly this, so you can quantize most of the model while leaving those layers native:

# leave every sliding window attention layer at native dtype
vllm serve <model> \
  --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers sliding_window
 
# or skip specific layer indices
vllm serve <model> \
  --kv-cache-dtype fp8 \
  --kv-cache-dtype-skip-layers 0 1 23

Worth knowing: with the Flash Attention 3 backend and FP8 KV cache, attention itself runs in the quantized domain, with queries quantized to FP8 alongside keys and values. That is a performance win and another reason to validate quality on your own evals rather than assuming the published regression numbers transfer.

The honest framing on FP8 accuracy: published regressions on long context retrieval benchmarks tend to be small but non zero, and they vary by model and task. Treat it as a change that needs an eval run, not a free lunch. In practice most teams ship it because doubling capacity usually matters more than a fraction of a point on a retrieval benchmark, but you should be the one making that call with your own numbers.

Layer 4: Capacity Tiering

Once you have squeezed the GPU, the next move is to stop treating HBM as the only place a KV block can live. Blocks that are not being attended to right now can sit in CPU DRAM or on NVMe and come back when needed.

vLLM's OffloadingConnector extends the prefix cache by pushing completed KV blocks out to slower, larger tiers as they are produced, and promoting hits back to GPU on demand. Transfers use cudaMemcpyAsync and run asynchronously alongside model compute, so the overhead on CPU and GPU cores is minimal.

Single tier, CPU only:

vllm serve <model> \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "block_size": 64,
      "cpu_bytes_to_use": 1000000000
    }
  }'

For multiple tiers you set spec_name to TieringOffloadingSpec and supply an ordered secondary_tiers list, consulted tier 0 first. One architectural constraint shapes everything about this: only the CPU primary tier has direct GPU access, so every transfer between GPU and a secondary tier is staged through CPU.

The unit of movement is a chunk, which by default maps to one accelerator block, and blocks_per_chunk lets you use larger chunks for bigger, more efficient I/O to host and secondary tiers. There is also per request control: setting max_load_tokens in kv_transfer_params caps how many tokens a request will pull from offloaded storage, with 0 disabling external loading entirely while leaving GPU prefix cache reuse intact. That is a useful knob for protecting latency sensitive requests from a slow tier fetch.

SGLang's equivalent is HiCache, with a three tier model: L1 on GPU, L2 in host memory, L3 in an external storage backend. The flags are concrete:

python -m sglang.launch_server --model-path <model> \
  --enable-hierarchical-cache \
  --page-size 64 \
  --hicache-ratio 2 \
  --hicache-io-backend kernel \
  --hicache-mem-layout page_first \
  --hicache-write-policy write_through \
  --hicache-storage-backend mooncake

--hicache-ratio 2 sets host memory to twice GPU memory, and --hicache-size in GB overrides it when set. The documented pairings matter: use page_first with the kernel I/O backend and page_first_direct with direct, since under direct a page_first layout silently becomes layer_first. Storage backends include hf3fs, mooncake, file, nixl, and aibrix, plus a dynamic loader for out of tree backends. And --hicache-storage-prefetch-policy trades latency against reuse, with wait_complete giving higher reuse than best_effort.

The payoff on the right workload is large. SGLang's published HiCache multi turn numbers show, with an external L3 tier, hit rate near 98% and effective input throughput of 145.5K tokens per second on DeepSeek-V4-Flash, against 9.4K for GPU only and 14.3K for GPU plus host. On a smaller model they report 96.8% hit rate and 67.1K tokens per second against 15.5K GPU only. Those are the project's own benchmarks on their chosen workloads, so treat them as directional, but the shape of the result is consistent with the Mooncake findings: for multi turn workloads with long shared prefixes, the cache tier is the dominant performance variable.

Layer 5: Topology

The last layer is about where the cache lives relative to your serving instances.

Shared cache pools. Instead of every replica keeping a private cache, a distributed pool lets any instance hit a block another instance computed. The vLLM and Mooncake Store integration is the clearest current example. A cluster wide master manages block metadata, service discovery, and dead node cleanup, while clients embedded in each GPU worker manage local DRAM and SSD and move blocks over RDMA. On the scheduler side vLLM hashes prompt token blocks and queries the master for matches. On the worker side GPU KV memory is registered as RDMA buffers using GPUDirect RDMA, which avoids staging through CPU memory and consumes no GPU SMs, with RDMA running on a dedicated background I/O thread.

Their reported results on real Codex traces with prefill and decode disaggregation across 12 GPUs: 3.8 times throughput, 46 times lower P50 TTFT, 8.6 times lower end to end latency, with hit rate going from 1.7% to 92.2%. Scaling from 12 to 60 GPUs with plain round robin routing, hit rate stayed above 95% with near linear throughput scaling. That last detail is the interesting one, because it means a large enough shared pool reduces how much your router has to be cache aware.

Prefill and decode disaggregation. Running prefill and decode on separate instances lets you tune TTFT and inter token latency independently, and gives you control over tail ITL that is hard to achieve by picking a chunked prefill size. vLLM supports it through a family of KV connectors including NixlConnector, LMCacheConnectorV1, MooncakeConnector, and FlexKVConnectorV1, and MultiConnector chains several together, which is how the Mooncake setup combines a PD connector with a store connector.

The vLLM docs include a warning that is easy to miss and important: disaggregated prefill does not improve throughput. It is a latency and tail control technique. If throughput is your goal, the wins are in layers 2 through 4, not here.

The Agentic Workload Case

If you serve coding agents or any multi turn agentic traffic, this section is the one that matters most, because the workload shape is unusually favorable to cache optimization and unusually punishing without it.

The vLLM Mooncake analysis characterized 610 real Codex traces from SWE-bench Pro work, with a median of 33 turns each. The numbers are striking:

• Input to output token ratio of roughly 131 to 1
• Context growing to about 80K tokens by turn 30, exceeding 180K in the longest traces
• Each turn adding only a few hundred to a few thousand new tokens
• Achievable cache hit rate of 94.2%
• A 100K token context taking about 3.8 GB of KV cache on their setup

Read those together and the implication is unavoidable. An agent turn is almost entirely prefix. If you recompute that prefix every turn you are paying for 80K tokens of prefill to generate a few hundred tokens of output, which is a catastrophically bad ratio. If you cache it, you pay for the delta. The same analysis notes that local offloading struggles here specifically because of eviction pressure and cross instance misses when the router moves a session, which is the argument for a shared pool rather than bigger per instance caches.

Practical consequence: for agentic serving, cache hit rate is not a secondary metric, it is close to being the primary one. A drop from 90% to 60% hit rate is roughly a four times increase in prefill compute.

How to Tell Which Layer Is Your Bottleneck

Do not optimize blind. Each symptom points at a different layer.

The metrics to actually collect:

Enable the residency histograms explicitly, since --kv-cache-metrics gates them. Metric names do drift between versions, so confirm against your deployment's /metrics endpoint rather than trusting any list, including this one.

Three Tuning Recipes

Starting points, not final answers. Measure on your own traffic.

Interactive chat, short prompts, latency sensitive. Prefix caching on for the system prompt, FP8 KV cache, context set to the real p99 rather than the model maximum, small batched token budget to protect inter token latency, session affinity in the router so turn two hits turn one's cache.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 2048

Long document RAG, big shared prefixes, throughput oriented. This is the best case for prefix caching, since many queries hit the same document. Push the batched token budget up, add a CPU tier so documents stay cached between query bursts.

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 16384 \
  --kv-transfer-config '{
    "kv_connector": "OffloadingConnector",
    "kv_role": "kv_both",
    "kv_connector_extra_config": {
      "block_size": 64,
      "cpu_bytes_to_use": 200000000000
    }
  }'

Coding agents, multi turn, very long contexts. Everything above plus a shared cache pool and cache aware routing, because this is the workload where instance local caching provably fails. Expect to run prefill and decode disaggregated, and expect hit rate to be your headline metric.

Pitfalls

• Prefix caching enabled, routing ignored. The most common cause of a disappointing hit rate. A per instance feature cannot fix a cross instance routing problem.
• Advertising a context length you never serve. It costs you scheduling headroom for no benefit.
• Uncalibrated FP8. Default scales of 1.0 are the version of this feature that gets blamed for quality loss. Calibrate with llm-compressor.
• Quantizing sliding window layers. They are more sensitive. Skip them explicitly.
• Expecting APC to speed up generation. It is a prefill optimization. Long output workloads see little from it.
• Expecting PD disaggregation to raise throughput. The docs say plainly that it does not. It is for TTFT and tail ITL.
• Treating a cold cache as a steady state. Hit rate after a rolling restart is not your real hit rate. Measure both, and consider cache warming.
• Offloading without measuring the slow path. A tier that is fetched often but slowly can be worse than recomputing. Watch promote latency against your prefill cost.
• Pinning --kv-cache-memory and forgetting it. It skips profiling, so it silently caps concurrency and breaks when the hardware or co tenancy changes.
• Benchmarking with synthetic single turn prompts. Agentic traffic has a 131 to 1 input to output ratio and enormous shared prefixes. A synthetic benchmark with unique prompts measures the one scenario where none of this helps.

Conclusion

KV cache optimization is not one trick, it is a stack, and the layers multiply. Architecture gives you up to an eight times advantage before you start. Correct sizing recovers memory you are currently wasting. Prefix caching with routing that actually lands on the right instance can take a workload from recomputing 80K tokens every turn to recomputing a few hundred. FP8 doubles what fits. Tiering to CPU and NVMe multiplies your effective cache far past HBM. A shared pool turns a fleet of private caches into one.

If you only do three things, do these: set your context length to your real traffic rather than the model's maximum, turn on FP8 KV cache with calibrated scales, and verify that your prefix cache hit rate is actually high in production rather than merely enabled in config. The Mooncake result of 1.7% to 92.2% hit rate on identical software is the most useful number in this entire post, because it shows how much of this problem is configuration and topology rather than hardware.

And measure on your own traffic. Every number here came from someone else's workload. The formula at the top, though, is yours to run, and it will tell you more about your deployment's ceiling than any benchmark will.

References

1. vLLM docs: Optimization and Tuning
2. vLLM docs: Automatic Prefix Caching
3. vLLM docs: Quantized KV Cache
4. vLLM docs: KV Offloading Usage Guide
5. vLLM docs: Disaggregated Prefilling
6. vLLM docs: Production Metrics
7. vLLM blog: Serving Agentic Workloads at Scale with vLLM and Mooncake
8. SGLang docs: HiCache Best Practices
9. LMSYS: Unified Radix Cache: One Tree for Hybrid Model Prefix Caching
10. LMSYS: SGLang HiCache: Fast Hierarchical KV Caching
11. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
12. LMCache: Technical report