Loss scaling is the technique of multiplying a training loss by a scalar factor to increase the magnitude of gradients during numerical computation.
Loss scaling multiplies a training loss by a scalar so gradients become large enough to survive limited numerical precision. Mixed-precision training commonly uses it because float16 gradients may round to zero. section 18.3.4
Use loss scaling in two practical ways:
Set a fixed factor with an optimizer's loss_scale_factor argument. section 18.3.4
Wrap the optimizer in LossScaleOptimizer to adjust the scaling automatically as training changes. section 18.3.4
In YOLO, scale box-placement errors by 5 and empty-cell confidence errors by 0.5 so important signals are not drowned out. section 12.2.4
Deep Learning with Python, Third EditionA technique that multiplies the loss by a scalar factor to increase gradient magnitudes and reduce underflow in low-precision training.
gradientproduces · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
objective functionconfigures · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.
