IBM-backed llm-d 1.0 adds controls for in-house AI inference

The open-source llm-d project, led by IBM, Red Hat and Google, released version 1.0 of its distributed AI inference framework for Kubernetes. It adds multi-tenant QoS, zero-downtime upgrades and cache-aware routing; IBM and Red Hat ran 3,000 agents at 5–10x lower cost than a hosted service.

• Version 1.0 adds QoS, zero-downtime upgrades and AI-agent serving patterns
• Router captured 97% of available cache reuse in an agentic coding test
• 3,000 agents ran concurrently at 5–10x lower cost than hosted serving
• Backers include IBM, Red Hat, Google, Nvidia, AMD, Intel and Mistral