DeepSeek DeepInfer Open Source Release Cuts LLM Inference Costs

DeepSeek open-sourced DeepInfer, its inference framework for production LLM serving, on October 20, 2026. The release targets cost reduction for operators running DeepSeek-V4.1-Flash at scale and provides core kernels, runtime components and scheduling tools under an Apache 2.0 license.

DeepSeek DeepInfer Technical Scope and Performance

The repository deepseek-ai/DeepInfer contains offline inference components only. Supported elements include optimized kernels such as FlashMLA and DeepGEMM, parallelism strategies for attention and mixture-of-experts workloads, DSpark speculative decoding with a default draft length of five, and structured output generation via regex or JSON Schema. The framework currently serves one model, DeepSeek-V4.1-Flash, with a 1-million-token context window.

Hardware requirements specify NVIDIA Blackwell SM100 GPUs or Huawei Ascend 950 NPUs running Linux with Python 3.12. NVIDIA users need CUDA 13; Ascend users require CANN 9.2. The code builds on vLLM concepts and incorporates elements from SGLang and TensorRT-LLM. Missing pieces for full online serving include KV cache management, prefill-decode disaggregation and cross-replica load balancing.

Self-reported production metrics indicate a single cluster processes roughly 155 trillion tokens per day with a 96 percent input cache hit rate. At off-peak pricing the system generates about 38 dollars of revenue per GPU-hour. On four SM100 GPUs for prefill and 64 for decode at 131k context length, the setup achieves over 25,000 tokens per second per GPU at high concurrency when DSpark is active. Time-to-first-token averages 400 milliseconds and sustained generation reaches 230 tokens per second.

Context, Community Response and Future Updates

DeepSeek has previously released model weights, kernels and tooling to reduce compute and energy expenses across the industry. The new framework continues that pattern. The vLLM project acknowledged the contribution and stated plans to integrate selected techniques into its own codebase.

Early GitHub activity shows thousands of stars within days of the October 16 repository creation. A companion MIT-licensed repository, deepseek-recipe, supplies token encoding and decoding routines that match the official API. Independent developers and smaller teams can now test the kernels without licensing fees, potentially lowering barriers for organizations that previously relied on proprietary stacks.

DeepSeek indicated that inference code for upcoming models and further kernel optimizations will appear in the same repository. Community members expect third-party projects to add the missing online-serving layers, while several research groups have announced plans for independent throughput and latency benchmarks on additional hardware platforms.