Loss scaling multiplies a training loss by a scalar so its gradients become large enough to survive low-precision arithmetic. In mixed-precision training, float16 gradients can be rounded down to zero; scaling makes that less likely without changing the intended optimization direction. section 18.3.4
Fixed scaling with Keras
Configure an optimizer with a fixed factor when a known value is appropriate: loss_scale_factor=10. The optimizer applies the scaling during training. section 18.3.4
optimizer = keras.optimizers.Adam(learning_rate=1e-3, loss_scale_factor=10)
Use a fixed loss_scale_factor directly in a Keras Adam optimizer. section 18.3.4
optimizer = keras.optimizers.LossScaleOptimizer( keras.optimizers.Adam(learning_rate=1e-3) )
Prefer LossScaleOptimizer when the appropriate factor may change during training; it adjusts scaling automatically. section 18.3.4
Do not confuse numerical loss scaling with task-specific loss weighting:
- YOLO scales the box-placement loss up by five and empty-cell confidence loss down by two, balancing competing detection objectives rather than preventing float16 underflow. section 12.2.4
- For mixed precision, use automatic scaling when possible because the useful factor can change over the course of training. section 18.3.4
