DeepSeek open-sources its DeepInfer inference engine

DeepSeek released its internal inference engine DeepInfer under Apache 2.0 on October 20, and it reached 19,941 GitHub stars in three days. The first version serves only DeepSeek-V4.1-Flash in batch mode and runs on Nvidia SM100 GPUs with CUDA 13 and Huawei Ascend 950 with CANN 9.2.

• Repository hit 19,941 stars and 2,126 forks in three days
• Supports Nvidia SM100 (CUDA 13) and Huawei Ascend 950 (CANN 9.2)
• Claims 155 trillion tokens per day per cluster and 230 tokens/s generation
• Missing KV cache, read/generation phase split and load balancing