Prime Inference splits prefill and decode across separate GPU pools

Prime Intellect launched Prime Inference, an open-source serving stack for open models on GB200 NVL72 racks built on NVIDIA Dynamo, vLLM, Mooncake and FlashInfer. Running prefill and decode on separate GPU pools cut p90 inter-token latency by nearly 40%, while a better memory layout reduced KV-cache transfer from 146ms to 78ms.

• Platform processed nearly a trillion tokens a day before launch
• p90 inter-token latency cut by nearly 40%
• KV-cache transfer reduced from 146ms to 78ms
• Stack is fully open source: Dynamo, vLLM, Mooncake, FlashInfer