Gemma 4 on SageMaker: QAT weights decode 2.05x faster than bf16 on one L4

A test on an Amazon SageMaker ml.g6.xlarge endpoint with one NVIDIA L4 showed the Gemma 4 E2B QAT checkpoint (4-bit weights) decodes at 105.1 tok/s versus 51.3 for bf16, and serves 1077.25 tok/s at 16 parallel requests versus 619.1. Both scored the same on 40 checked questions, while 4-bit weights save 18% of GPU memory.

• QAT checkpoint decodes at 105.1 tok/s vs 51.3 for bf16 on one L4
• At 16 parallel requests: 1077.25 tok/s vs 619.1
• 4-bit weights use 8.01 GiB vs 9.75 GiB, an 18% saving
• Identical score on 40 checked questions