FP16 quantization quality loss traced to one overflowed tensor, not architecture
A post-mortem found that LaMa's fp16 quality collapse came from 16 BatchNorm running_var tensors exceeding fp16's 65,504 ceiling and being silently clamped by the converter, not from mask-gated architecture. A two-line fix restores fp32-level quality with a 52% smaller download.
- 16 of 606 running_var tensors exceed fp16's 65,504 limit
- Rounding mantissa to 10 bits cost only 0.004 dB versus fp32
- Unfixed fp16 model scores 15.8 dB against fp32's 29.7 dB
- After the fix the model matches fp32 and is 52% smaller
Read next
AI