Cross-entropy loss measures the expected negative log probability that a model assigns to outcomes drawn from a target distribution. For a target distribution q and model distribution p, H(q,p) = -Σz q(z) log p(z): predictions are penalized when they assign low probability to outcomes the target considers likely. section 12.3.1
In language-model training, the target distribution determines what the loss teaches:
- Standard one-hot next-token training focuses on whether the single target token receives probability. section 12.3.1
- Knowledge distillation uses soft teacher probabilities over the vocabulary, training a student distribution to match the teacher at every position. section 12.3.1
- Supervised fine-tuning minimizes a loss equivalent, up to a constant, to forward KL divergence from the data distribution to the model. section 15.2.2
Per-step loss in a process reward model
A process reward model predicts whether each reasoning step is incorrect, neutral, or correct, but computes cross-entropy only at designated step boundaries. Other token positions receive the mask value -100 and do not contribute to the loss. A binary variant instead sums the negative log-likelihood of each step’s validity label. section 5.5
Equation 12.3
\[\label{eq:kd_cross_entropy} H(q,p) = -\sum_z q(z)\log p(z).\]The compact equation exposes the two distributions, the expectation over target outcomes, and the negative log probability minimized during training. section 12.3.1
