HPE Alletra X10000 speeds inference up to 20x with KV cache offload

HPE is using its Alletra Storage MP X10000 to hold KV cache that would otherwise be evicted from expensive GPU memory and recomputed. In Kamiwaza-validated tests on Nvidia H200 GPUs with Nemotron 70B and over 30 concurrent agents, time to first token improved up to 20x and effective inference throughput rose 17x.

• Tests used Nvidia H200 GPUs, Nemotron 70B and 30+ concurrent agents
• Up to 20x faster time to first token with KV cache offload
• Effective inference throughput up to 17x higher
• Gains depend on context length, concurrency and cache hit rates