Running a 35B Parameter AI Model on iPhone at 11 Tokens per Second

Better Stack demonstrated running a 35-billion-parameter MoE model directly on an iPhone: only 3 billion parameters are active, at 11 tokens per second. Three-level quantization compressed the model from 19 to 13 GB, active components take 1.4 GB RAM, and inactive experts (12 GB) are streamed from SSD.

• MoE architecture activates 3 billion of 35 billion parameters via router
• Quantization compressed model from 19 to 13 GB: hot experts 4-bit, cold 2-bit
• Active components take 1.4 GB RAM, 12 GB of experts stored on SSD
• Generation speed is 11 tokens per second on iPhone