Developer reproduces softmax and cross-entropy numerical bugs in NumPy
A developer verified on NumPy 1.26.4 a paper claiming two numerical bugs in standard softmax and cross-entropy implementations. In fp32 the loss and target gradient round to zero at a logit margin near 16, and distillation gradients hit an error floor of about 4.6e−08.
- The fp32 zeroing occurs at margins of 15.8–16.7 depending on the reduction tree
- At m=15 the standard loss reads 2.38e−07 instead of 3.06e−07, a 22% error
- In distillation the standard gradient is 100% wrong in one component; the fixed version errs at 9.2e−16
- The expm1/log1p fix matches the fp64 reference eight orders of magnitude closer
Read next
Software