vLLM Semantic Router Decision 3.0 tested on local hardware

The open-source Decision 3.0 model from the vLLM Semantic Router project skips text generation and returns calibrated Choice, Yes/No and Score probabilities in a single forward pass. On a single 24GB GPU, the 9B BF16 build used 15.64GB of VRAM and answered in 239.5ms, while the 27B 4-bit NF4 build used 14.33GB and took 808.6ms.

• 9B d3-flash in BF16: 15.64GB VRAM, 239.5ms for three text questions
• 27B in 4-bit NF4: 14.33GB VRAM, 808.6ms, 15.49s cold start
• On a pneumothorax X-ray, 27B flagged STAT at 98.79% vs 88.46% for 9B
• Suggested split: 9B for fast triage, 27B for high-risk decisions