Ant Group and vLLM open-source Humming for quantized LLM inference

Ant Group and the vLLM community have released Humming, an Apache 2.0 JIT-compiled GEMM kernel library for quantized LLM inference. It covers INT1–INT8 and FP4–FP8 weights, dense and MoE GEMM, Nvidia GPUs from Turing onward, and T-Head's Zhenwu 810E accelerator. Humming ships as a dependency in CUDA builds of vLLM starting with 0.29.0.

• Supports INT1–INT8 and FP4–FP8 weights with FP16, BF16, FP8, FP4, INT8 or INT4 activations
• Dense W4A16 reported at roughly 3–4x cuBLAS on small batches
• MoE W4A8 on eight H20 GPUs raised throughput by about 14–43%
• Pip package is about 100 KB and depends only on PyTorch and NVCC