Push-based KV transfer in vLLM cuts TTFT by up to 3x for disaggregated inference
“push mode delivered up to 3x better TTFT and up to 30 percent better time-per-output-token (TPOT) compared to pull mode”
KV cache
8 tracked signals on KV cache.
AWS contributes bi-directional KV cache transfer to vLLM, cutting TTFT up to 3.3x
Agentic AI demands a new enterprise data category—agent context—forcing storage to become systems of context.
“Storage needs to evolve. It needs to move from being what we've always known as a system of record, to now a system of context.”
Most vLLM deployments fail production due to guessed configs, not model quality.
“I've seen a lot of cases where everyone is just guessing.”
Red Hat demonstrates KV cache-aware routing and prefill/decode disaggregation for agentic LLM workloads on Kubernetes.
“benchmarks actually don't show you is the chaotic reality of multi-turn interactions, massive context fluctuations which are very typical of agentic workloads”
AWS tiered KV cache on SageMaker HyperPod delivers 2.7x TTFT improvement and 100% cross-Pod cache hit rate
“With this architecture, workloads that previously required P5 instances can run on lower-cost G6e instances, reducing per-endpoint cost.”
NVIDIA Vera delivers faster encryption, compression, and recovery for agentic AI storage workloads.
“Storage is an active part of every agentic AI workflow.”
LLM inference is bottlenecked by memory bandwidth, not compute speed
“LLM inference is actually limited by memory bandwidth, not computational speed.”