vLLM push KV transfer cuts first-token latency threefold

Engineers from AWS and Mistral added a push connector to vLLM that transfers the KV cache between prefill and decode instances. On EC2 P5en at 8 queries per second with 16,000-token inputs, time to first token fell from about 1,744 ms to about 540 ms.

• TTFT dropped from ~1,744 ms to ~540 ms at 8 QPS with 16,000-token inputs
• Decode instance pre-registers memory while prefill is still computing
• Transfers run on a background nixl-push-writer thread, off the forward pass
• Benchmarks ran on AWS EC2 P5en with EFAv3 at 3,200 Gbps and GPUDirect RDMA