On October 20, 2026, at the PyTorch Conference North America in San Jose, California, IBM, Red Hat and Google announced llm-d 1.0. The release marks the first production-ready version of the open-source, Kubernetes-native framework for distributed LLM inference. Contributors positioned the update as a response to rising inference costs and the need for reliable multi-tenant deployments.
Production Features Added in llm-d 1.0
The 1.0 release introduces Day 2 operational controls that address stability and resource sharing. Multi-tenant QoS now includes priority and fairness-aware flow control along with probabilistic admission. Router high availability supports hot standby, while rolling updates and DisaggregatedSet enable coordinated prefill and decode upgrades without downtime. A stable v1 configuration API and agent-aware serving with session routing and turn-priority scheduling complete the set of hardened capabilities.
Technical enhancements cover prefix- and load-aware routing, KV cache offloading and tiering, prefill and decode disaggregation, wide expert parallelism, and batch or asynchronous inference modes. The framework integrates with engines such as vLLM and SGLang and runs on NVIDIA, AMD or Intel GPUs as well as Google TPUs. It supports any cloud or on-premises Kubernetes cluster through a vendor-neutral interface.
Red Hat packages llm-d 1.0 inside Red Hat AI for OpenShift and managed Kubernetes environments. This packaging allows enterprises to adopt the stack without building custom orchestration layers. The project entered CNCF Sandbox status earlier in 2026 and now lists more than 500 contributors from over 25 organizations.
Measured Performance and Early Adoptions
Google integrated llm-d 1.0 into the GKE Inference Gateway for both TPUs and GPUs. Internal tests in Vertex AI production showed a greater than 35 percent reduction in time to first token and a doubling of prefix-cache hit rates from 35 percent to 70 percent. The setup also demonstrated multi-cluster scaling for larger workloads.
IBM and Red Hat reported running approximately 3,000 concurrent software-engineering agents on 544 H100 GPUs at roughly $0.11 per million tokens. This cost sits between five and eleven times lower than comparable proprietary services, with prefix-cache hit rates reaching 97 to 98 percent. Tesla and Red Hat recorded three times higher output throughput and twice faster time to first token on Llama 3.1 70B using AMD MI300X accelerators.
Additional adopters include DigitalOcean, Mistral AI, JetBrains and Prime Intellect. These organizations cite the framework’s support for agentic workloads and self-hosted inference as primary drivers. Hyperscalers and neocloud providers have begun embedding the stack in their managed offerings to meet demand for cost-efficient, hardware-agnostic serving.
Next Steps for Broader Deployment
CNCF governance continues with community-led enhancements focused on additional hardware support and tighter integration with existing orchestration tools. Organizations evaluating self-hosted inference can start with the stable configuration API and test multi-tenant policies on existing clusters before full rollout.
Red Hat customers already running OpenShift gain immediate access through the Red Hat AI packaging. Teams using GKE can enable the Inference Gateway integration to capture the reported latency and cache improvements. Continued production deployments across model labs and enterprises are expected as the hardened stack matures for agentic and regulated workloads.
Frequently asked questions
What is llm-d 1.0?
llm-d 1.0 is the first production release of an open-source Kubernetes-native framework for distributed LLM inference. It was announced on October 20, 2026, by IBM, Red Hat and Google and adds Day 2 operational controls such as multi-tenant QoS and zero-downtime updates.
Which hardware does llm-d 1.0 support?
The framework runs on NVIDIA, AMD and Intel GPUs as well as Google TPUs. It works in any cloud or on-premises Kubernetes environment through a vendor-neutral interface.
How does llm-d 1.0 reduce inference costs?
IBM and Red Hat reported running 3,000 concurrent agents on 544 H100 GPUs at about $0.11 per million tokens, five to eleven times lower than proprietary alternatives, while achieving 97–98 percent prefix-cache hit rates.
Where can users obtain llm-d 1.0 today?
Red Hat includes the release in Red Hat AI for OpenShift users. Google offers integration through the GKE Inference Gateway. The source code and documentation are available on the official project site.
By YasTech Editorial
Comments 5