Loss scaling
Loss scaling is a low-precision training technique that multiplies the loss by a scalar factor to increase gradient magnitudes and reduce numerical underflow.
Deep Learning with Python, Third Edition
A technique that multiplies the loss by a scalar factor to increase gradient magnitudes and reduce underflow in low-precision training.
- How it is used
- Uses scalar loss multiplication to preserve small gradients during mixed-precision training.
- How deep it goes
- Explains gradient underflow and gives fixed and automatic Keras configurations, but does not derive scaling internals.
- Examples
- Uses Keras optimizer snippets and contrasts them with YOLO’s task-specific box and confidence loss weights.
- What you can do after
- Operate/configure mixed-precision training with fixed or automatically adjusted loss scaling in Keras.
Read how Deep Learning with Python, Third Edition explains loss scaling →
Deep Learning with R, Third Edition
Loss scaling is the technique of multiplying a training loss by a scalar factor to increase the magnitude of gradients during numerical computation.
- How it is used
- Protects small gradients during mixed-precision training and rebalances component errors in YOLO object detection.
- How deep it goes
- Explains the numerical motivation and optimizer choices, then applies fixed factors to YOLO loss components without deriving scaling internals.
- Examples
- Uses Keras optimizer configurations and YOLO factors of 5 for box errors and 0.5 for empty-cell confidence errors.
- What you can do after
- Configure loss scaling through a fixed optimizer factor or an automatic LossScaleOptimizer wrapper.
Read how Deep Learning with R, Third Edition explains loss scaling →
Related concepts
- gradientproduces · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- objective functionconfigures · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.