Mixed precision speeds model computation by using 16-bit values where precision is safe while retaining 32-bit values for numerically sensitive work. During training, most forward-pass operations use float16, while weights remain float32 and receive float32 updates. section 18.3.3
Turning on mixed precision
Enable Keras mixed precision with `keras.config.set_dtype_policy("mixed_float16")`. Most layers then compute in float16 while retaining float32 variables. Opt out individual unstable layers, such as softmax or crossentropy, by setting `dtype="float32"`. section 18.3.3
Keep these practical rules in mind:
- Use float32 for gradients and variable updates because float16 cannot reliably represent common small updates such as 1e-5 or 1e-6. section 18.3.3
- Scale the loss when small float16 gradients risk rounding to zero; prefer `LossScaleOptimizer` when the appropriate scale changes during training. section 18.3.4
- Expect speed benefits on GPUs or TPUs with little model-quality impact, while treating numerically unstable operations as exceptions. section 18.3.3 section 18.4
import keras keras.config.set_dtype_policy("mixed_float16")
Inspect the minimal Keras configuration that enables the mixed-float16 policy. section 18.3.3
optimizer = keras.optimizers.Adam(learning_rate=1e-3, loss_scale_factor=10)
Inspect a fixed loss-scale setting attached directly to an Adam optimizer. section 18.3.4
optimizer = keras.optimizers.LossScaleOptimizer( keras.optimizers.Adam(learning_rate=1e-3) )
Inspect the recommended automatic loss-scaling wrapper around an optimizer. section 18.3.4
