Mixed precision uses lower-precision formats for some computations while retaining higher precision for others. In training, this can reduce memory use and speed computation without the numerical risks of using only 16-bit arithmetic. section 6.1
A typical mixed-precision training workflow:
- Convert weights to lower precision for faster computation and calculate gradients at lower precision. section 6.1
- Convert gradients back to higher precision for numerical stability, then update the original weights using scaled gradients. section 6.1
For inference, pure FP16 can use less memory but may degrade performance. Combining FP16 and FP32 uses more memory than pure FP16, but less than pure FP32, with comparable or sometimes slightly better performance. section 6.1
Estimating inference memory
A 2.7-billion-parameter GPT-Neo model is estimated to require 12.96 GB in FP32 and 5.4 GB in FP16. These estimates show how precision choice affects whether a model fits available GPU memory. section 4.2


