Google Cloud ships llm-d 1.0 with cache-aware routing for TPU and GPU fleets

Google Cloud released llm-d 1.0, the first stable version of its open-source Kubernetes-native distributed inference stack built on vLLM and SGLang. On a 17,000-node multi-region GKE benchmark, median time-to-first-token dropped over 85% and throughput rose 2.4x on an Ironwood TPU workload.

MarkTechPost

Open source

Google Cloud ships llm-d 1.0 with cache-aware routing for TPU and GPU fleets

Median TTFT cut by over 85%, throughput up 2.4x on Ironwood TPU coding workload
Backed by Red Hat, IBM and Google with 500+ contributors from 25+ organizations
New: multi-tenant QoS, zero-downtime upgrades, async inference via Pub/Sub and Redis
Supports NVIDIA, AMD, Intel GPUs and TPUs under Apache 2.0, CNCF Sandbox status

#Google#Llm-d#VLLM#SGLang

Google Cloud ships llm-d 1.0 with cache-aware routing for TPU and GPU fleets

QAT Gemma 4 26B-A4B on one TPU v6e: 15.6x KV cache, 1.9x throughput vs FP8

OpenAI and Microsoft launch typed decision APIs for AI agents

IBM-backed llm-d 1.0 adds controls for in-house AI inference

Repacked QAT Gemma 4 on one TPU v5e: 12B serves at 675 tokens/s