Cross-entropy loss measures prediction error by taking the negative logarithm of the probability assigned to the correct next token. Higher confidence in the right answer lowers the loss; confidence in the wrong answer raises it sharply. section 11.30
For each token, compare the model’s probability for the actual next token with the ideal probability of 1. A perfect prediction has loss 0 because -log(1) equals 0. section 11.30
The same correct answer produces different losses depending on confidence:
- Confident and correct: p = 0.99 produces a tiny loss of about 0.01. section 11.30
- Uncertain: p = 0.50 produces a moderate loss of around 0.69. section 11.30
- Confident and wrong: p = 0.01 produces a huge loss of about 4.60, heavily penalizing misplaced confidence. section 11.30
Table 11.34 A subset of tokens and their loss values
|
ID |
Token |
Correct next ID |
Correct next token |
Probability |
Loss |
| 0 |
a |
1 |
l |
0.0243 |
3.7221 |
| 1 |
l |
2 |
ic |
0.0395 |
3.2304 |
| … |
|||||
| 30 |
ing |
6 |
b |
0.1123 |
2.1869 |
Browse token IDs, correct next tokens, assigned probabilities, and corresponding loss values. section 11.30
