vLLM Blog · September 8, 2026 · 24 min read
Source: https://vllm.ai/blog/2026-09-08-vllm-agentx
TL;DR: Agentic workloads are becoming a major source of vLLM traffic. Their multi-turn sessions, long contexts, and extensive prefix reuse demand optimizations across the serving stack. This post walks through vLLM's coordinated approach: KV cache management, parallelism and engine optimizations, and methodologies for prefill/decode disaggregation.
Measured on AgentX, SemiAnalysis's public agentic benchmark, vLLM achieves up to 130K total tokens per GPU-second on DeepSeek V4 Pro, and an interactivity of up to 376 tokens per second on MiniMax M3. Across DeepSeek V4 Pro, MiniMax M3, and Kimi K3, vLLM delivers a 14.6×–106× serving-cost advantage over Opus 5 API pricing (see Performance).
Characterizing agentic workloads: a second look
Since our first post on serving agentic workloads in May, the share of agentic traffic has continued to grow. As of June 2026, OpenAI reported that Codex generated 64% of combined Codex and ChatGPT output tokens among enterprise customers.
This growing token consumption stresses serving infrastructure along two axes: cost and latency. Cost efficiency determines how many concurrent agents fit in a fixed hardware budget; latency determines how quickly each agent progresses through its reasoning and tool-use cycles. Optimizing agentic serving therefore means improving the latency-cost frontier as a whole.
To evaluate that frontier under representative traffic, SemiAnalysis recently released AgentX, a public benchmark built from real-world agentic coding traces. These traces provide a concrete view of the workload characteristics that serving systems must accommodate:
- Long-running, multi-turn sessions. Median 43 turns per session.
- Long contexts with short outputs. Median input 142K tokens, median output 444 tokens.
- Extensive prefix reuse. Prefix-cache hit rate above 96%.
- Subagent-heavy traffic. 44% of sessions contain at least one subagent, with a median of four subagent rollouts among those sessions.
These statistics follow from how an agentic session is built. Each turn appends the latest tool result to the accumulated context and sends the whole thing back to the model, so the input keeps growing while each turn adds only a short new prefill, and almost all of the request is a prefix the engine has already seen. Subagents either fork from that context or start fresh, and their results are joined back into the parent before the final answer.
Challenges in serving agentic workloads
These workload characteristics create three challenges for efficient serving.
- Prefix cache pressure. Every turn of a multi-turn session replays the full conversation so far. To keep many sessions running at once, the engine has to offload KV caches between turns. This becomes harder at scale, where KV cache management, prefix caching, and offloading must work efficiently across GPUs, prefill/decode disaggregated instances, and replicas.
- Execution efficiency. Agentic workloads feature long contexts and tight latency requirements, so the engine has to process more tokens and do more work per token in less time. This requires adapting parallelism, kernels, scheduling, speculative decoding, and other engine optimizations to the new request shape.
- Finding the right P/D ratio. Context lengths and cache hit rates vary wildly across sessions and subagents, and routing must efficiently balance cache affinity and load across ranks. These factors make it difficult to find the throughput-optimal P/D ratio, which also shifts with concurrency.
The vLLM approach: optimizations across the stack
Figure 3 summarizes the three planes. The data plane manages a distributed shared KV cache, the execution plane maps each model to appropriate parallelism and kernels, and the control plane coordinates proper P/D ratio and request scheduling.
Hybrid KV cache management
KV cache management has been central to vLLM since PagedAttention, and agentic workloads with long contexts put heavier pressure on KV cache capacity. Modern hybrid models complicate allocation further by combining sliding-window and linear attention with full attention, whose cached blocks differ in size and lifetime.
vLLM's hybrid KV cache manager tackles this complexity with a simple core idea: a uniform memory page as the basic allocation unit, managed through one shared block pool. A shared pool lets vLLM reallocate memory dynamically on demand instead of statically partitioning capacity by attention type.
The new packed KV cache layout for DeepSeek V4 stores all cache groups and layers in one contiguous backing allocation per block rather than 92 fragmented ones. This reduces descriptor and P/D transfer overhead, and also permits a smaller allocation unit when the FP4 indexer is enabled, saving roughly 10% of KV cache memory.
Hierarchical KV cache offloading
To preserve prefix caches beyond GPU memory capacity and across each engine, vLLM has integrated Mooncake Store as a distributed KV cache pool. Hierarchical tiers extend capacity with disks and extra CPU-only nodes via Mooncake Store standalone-store mode.
Session-aware prefix-cache retention for hybrid models:
- Interval-based retention automatically preserves prompt-end caches/linear states at each turn.
- Marconi-style selective retention retains a checkpoint when a prefix is observed a second time.
Together, these policies preserve a high cache hit rate without excessive storage overhead on large-scale agentic workloads.
Model-specific parallelism
Kimi K3: MLA compresses KV into a single latent space; plain TP replicates that latent cache inefficiently. Decode context parallelism (DCP) shards the cache along the sequence dimension. DCP offers lower decode latency and higher throughput/KV capacity for agentic prefixes. Optimized DCP uses symmetric-memory buffers and fused kernels, cutting latency by about 13% per layer vs default DCP8. On NVL72-class systems, wide EP with data parallelism (DEP) can scale better than DCP at higher per-rank batch sizes.
DeepSeek V4: Compressed sparse attention makes TP head sharding compute-inefficient. Prefill context parallelism (PCP) performs best for long prefills (PCP8 achieves 2.65× prefill speedup over TP8 for a 32K prompt). DEP is the default for most DeepSeek V4 configurations. DCP did not transfer cleanly to V4 (see bitter lessons).
Scheduling mixed agentic traffic
Breaking head-of-line blocking: --long-prefill-token-threshold caps how many tokens one request may schedule per step. With a 512-token threshold on DeepSeek V4 Pro / B300s, TPGS rises by up to 93% and P90 interactivity improves ~2.3× (higher TTFT for the long request).
Align DEP prefill schedule cadence: --prefill-schedule-interval admits prefill work only every Nth engine step, aligned across DP ranks, so MoE all-to-all lockstep is not repeatedly stalled by staggered prefills.
Scaling with optimal P/D disaggregation
Two-phase rate-matching methodology:
Phase 1: Saturation profiling of prefill-only and decode-only deployments. Phase 2: P/D sweep from Phase 1 saturation points.
Performance: agentic-first and openly verifiable
Independent validation on SemiAnalysis AgentX — open dataset from $3M of real-world agentic coding traces with 1M context, public infrastructure of >1,000 chips / ~2 MW.
Highest-throughput vLLM configs maintaining P90 interactivity > 50 tok/s/user:
| Model | GPUs / concurrency | Total tokens per GPU-second (TPGS) @ P90 > 50 tok/s | P90 interactivity |
|---|---|---|---|
| DeepSeek V4 Pro 1.6T | 12 GB300s / 256 | 83K TPGS | 58.3 tok/s |
| MiniMax M3 428B | 2 B300s / 24 | 70K TPGS | 74.2 tok/s |
| Kimi K3 2.8T | 16 GB300s / 48 | 11.8K TPGS | 62.7 tok/s |
Serving-cost advantage vs Opus 5 (cached input × $0.50/M + uncached × $5/M + output × $25/M; theoretical cache hit favorable to Opus):
| Model | GPU TCO/hour | Equivalent Opus 5 cost/hour | Cost advantage |
|---|---|---|---|
| DeepSeek V4 Pro 1.6T | $27.72 | $2,926 | 106× |
| MiniMax M3 428B | $4.52 | $384 | 85× |
| Kimi K3 2.8T | $36.96 | $538 | 14.6× |
AgentX harness public at SemiAnalysisAI/agentx-harness; results on InferenceX dashboard.
The bitter lessons
- Pipeline parallelism does not fit warm agentic turns. PP/CPP work for cold compute-heavy prefills; most agentic turns are warm prefix-heavy with only hundreds–thousands of new tokens — pipeline bubbles dominate.
- DCP does not transfer cleanly to DeepSeek V4. Complex attention stack (indexer + compressor + main attention) makes CP coordination costly; after heavy investment DCP only matched DEP.
- Load balance does not guarantee better performance. Queue-depth / KV-utilization balancing underperformed session-aware sticky routing on AgentX: short inter-turn delays make cache locality more valuable than perfect load balance; async KV transfer still occupies destination capacity.
The path ahead
Make agentic structure explicit throughout the stack:
- Control plane: separate routing for first-turn long fresh prefills vs turn 2+ high-reuse appends.
- Agent hints from frameworks/harnesses (session structure, branching points, tool-call latencies, lifecycle) via standardized APIs.
- Programmable KV cache placement/retention/eviction.
- Session-based KV management with prefetch during idle inter-turn gaps.
Acknowledgments
Led by Inferact with extensive support from the vLLM community. Thanks to SemiAnalysis for AgentX; NVIDIA and AMD for collaboration.
Full original post including figures: https://vllm.ai/blog/2026-09-08-vllm-agentx