Compressing An 11B VLM To 2.7-bit Weights For Mobile CPUs

Graphcore Research and Arm introduced Llama-Mobile: the S3D8 format compresses Llama 3.2 Vision 11B weights from 21.3 GB to 3.7 GB (2.7 bits per parameter) and accelerates token generation on Arm CPUs. On Pixel 8a, the model achieves 3.8 tokens/s, retaining 66.1% versus 74.4% for bfloat16.

• Weights of Llama 3.2 Vision 11B compressed from 21.3 GB to 3.7 GB — over 80% savings
• S3D8 format stores 3 weights per byte: 5-bit centroid index and signs
• On Pixel 8a, token generation reaches 33.8 GMAC/s — 28% higher than INT8
• QAT distillation yields 66.1% in VQA versus 74.4% for bfloat16