Tiered KV Cache Offloading in vLLM

vLLM Blog · September 10, 2026 · 10 min read

Source: https://vllm.ai/blog/2026-09-10-tiered-kv-offloading

Summary (verbatim-leaning)

Long-context models and multi-turn conversations generate massive KV caches. When accelerator memory (e.g., GPU HBM) fills up, previously computed KV data is evicted. On the next request that needs it, vLLM must recompute it from scratch.

Tiered KV cache offloading preserves evicted KV data across host memory, storage, and remote peers. Instead of recomputing, vLLM reloads the data from a lower tier — saving compute, reducing latency, and increasing the effective serving capacity of the cluster.

With secondary tiers, KV data also becomes shareable across nodes — enabling horizontal scaling of the cache, warm-starting new instances from shared storage, and transferring KV data between peers for disaggregated serving or load balancing.

The framework has been available in vLLM since v0.22.

Key claims (verbatim-leaning English extract)

• Host-centric design: all KV data flows through host memory (CPU DRAM). Offload: accelerator → host → secondary tiers (filesystem, object storage, remote peers). Reload reverses the path.
• Fast accelerator release: PCIe copy to host frees HBM before secondary transfers; just-in-time allocation on reload.
• Consolidated I/O: TP shards fan into one shared host region → fewer, larger I/Os to secondary tiers.
• Canonical host layout: configuration-independent chunks; TP=2 and TP=4 nodes share without remapping; works across FlashAttention / FlashInfer / Triton backends.
• Secondary tiers are single-process CPU libraries (POSIX / S3 / RDMA); never touch accelerator APIs.
• Unit = chunk (default one accelerator block; blocks_per_chunk configurable). Host tier is LRU/ARC cache, not a staging buffer.
• Reload: host hit immediate; else query secondary in order; first HIT promotes to host; scheduler may RETRY.
• Tier types: fs (content-addressed files; shared mount = automatic sharing), obj (S3-compatible via NIXL), p2p (ZMQ + RDMA host-to-host; orchestrator e.g. llm-d chooses peer via kv_transfer_params).
• P2P use-cases: Prefill/Decode disaggregation; load balancing.
• Hybrid model support: full attention / sliding window / MLA / Mamba via uniform byte-buffer chunks (DeepSeek V4, GLM 5.3, Nemotron 3, …).
• Observability: Prometheus /metrics (host util, transfer throughput, per-tier latency/hit rate); structured KV events for llm-d / Dynamo routing and P2P orchestration.
• Extensibility: SecondaryTierManager with lookup / submit_store / submit_load / get_finished_jobs; out-of-tree via module_path.
• Scale narrative: ~64 conversations HBM OK; 64–128 need CPU offload; beyond 128 storage tier doubles throughput vs alternatives (benchmark: Qwen3.6-35B-A3B, 2×H100 TP=2, NVMe FS, multi-turn 12K+4K×8, concurrency 64).

Remainder

Full original English post (figures, config snippets, acknowledgements): see source_url.