Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
Push-based KV transfer in vLLM cuts TTFT by up to 3x for disaggregated inference
“push mode delivered up to 3x better TTFT and up to 30 percent better time-per-output-token (TPOT) compared to pull mode”
AWS contributed a push-based KV transfer mode to vLLM that moves post-prefill coordination off the critical path in disaggregated prefill-decode inference deployments. Benchmarked on EC2 P5en instances with EFA, push mode yields up to 3x TTFT improvement and 30% better TPOT versus the default pull mode for Qwen3-32B at high request rates. This is a meaningful open-source contribution to production LLM serving infrastructure, relevant to anyone optimizing inference throughput and latency at scale.