chiprook
← AI
AIOctober 18, 2026, 11:23

FP16 quantization quality loss traced to one overflowed tensor, not architecture

A post-mortem found that LaMa's fp16 quality collapse came from 16 BatchNorm running_var tensors exceeding fp16's 65,504 ceiling and being silently clamped by the converter, not from mask-gated architecture. A two-line fix restores fp32-level quality with a 52% smaller download.

FP16 quantization quality loss traced to one overflowed tensor, not architecture
#LaMa#ONNX
Read next
AI

laya-compactor cuts RAG context tokens by 70% with no answer-quality loss

AI

421M-parameter Laya model runs in a browser tab, download up to 478 MB

AI

DualQuant Quantizes LLMs to 4-bit Without Calibration Data

AI

Quantized LLMs recover 97% accuracy; IBM adds reasoning to Granite