llm-d 1.0 Targets 5–10x Lower AI Inference Costs

The open-source llm-d project, founded by Red Hat, IBM, Google Cloud, CoreWeave and Nvidia and now governed by CNCF, released version 1.0. In a benchmark on NVIDIA GPUs, the 753B-parameter GLM 5.2 model delivered over 6.6 million output tokens per minute and served up to 3,000 concurrent coding agents without preemptions, with inference costs 5–10x lower than commercial API pricing.

• 753B-parameter GLM 5.2 hit 6.6 million output tokens per minute on NVIDIA GPUs
• Supported up to 3,000 concurrent coding agents without preemptions
• Inference costs estimated 5–10x lower than commercial APIs
• IBM Research saw 11.5x lower costs with a 98% cache hit rate