KV cache moves to SSDs, but the drives still lack a standard

Huawei's OceanStor M900 and Nvidia's CMX Context Memory Storage both offload reusable inference KV cache to a separate shared tier, yet no agreed SSD specification exists for that data. Raising the KV cache hit rate from 90% to 95% halves recompute cost, but requires keeping far more historical data with lifetimes from minutes to weeks.

• Huawei OceanStor M900 and Nvidia CMX move KV cache to a shared tier
• Raising hit rate from 90% to 95% cuts recompute from 10% to 5%
• DRAM-to-SSD replacement runs at ratios of 1:N, sometimes 1:5 or 1:10
• Endurance needs shifted from 3 to 9 DWPD; convergence expected in 6-12 months