NaN, or “not a number,” appears when training produces an undefined or unrepresentable numerical result. A loss exploding toward NaN usually signals a computational or implementation problem rather than ordinary model behavior. section 9.1.3
When a loss trends toward NaN, check:
- Exploded gradients; try gradient clipping, a lower learning rate, or different weight initialization. section 9.1.3
- Mathematical errors such as division by zero or taking logarithms of zero or negative numbers. section 9.1.3
- NaNs already present in the data, often indicating missing preprocessing or an implementation error. section 9.1.3
A loss curve that breaks
A model may converge normally and then suddenly trend toward NaN. Treat the break as a debugging signal: inspect gradients, numerical operations, input data, learning rate, and initialization. section 9.1.3
Figure 9.4 A model was converging until something went wrong.
Use the figure to recognize a training curve that converges until a computational problem interrupts it. section 9.1.3
