120B Nemotron-3.5-Super runs on an 8 GB GPU

A developer ran Nvidia's 120B MoE model Nemotron-3.5-Super on a single 8 GB RTX 5060, streaming 59 GiB of experts from NVMe and reaching 8.2 tokens/s in Open WebUI. The setup matches the Hugging Face reference and also handles images via a streamed C-RADIO vision tower.

• The 120B NVFP4 model is 75 GB, but only 6.6 GiB of non-expert weights stay on the GPU
• Skipping zeros after ReLU² added 12%: 6.6 → 7.3–7.6 tokens/s
• A 20 GB RAM expert cache raised throughput to 8.2 tokens/s
• The 1.3 GB C-RADIO vision tower processes an image in 0.4–1.6 s