Cross-entropy loss trains classifiers by measuring how well predicted probabilities assign mass to the correct class. It converts class scores into probabilities, then penalizes the probability assigned to the target label. section 2.3.2
For C-class prediction, the model produces a score vector ŷ ∈ ℝC, while the target y identifies one class from {0, 1, …, C − 1}. Softmax turns the scores into C probabilities that sum to 1. Cross-entropy then compares that distribution with the target distribution, commonly one-hot: 100% for the correct class and 0% for every other class. A confident correct prediction receives low loss; assigning probability near zero to the true class produces high loss. section 2.3.2
The same loss idea appears in several training settings:
- Multiclass networks use cross-entropy to compare a vector of class predictions with one correct integer label. section 2.3.2
- GAN discriminators use binary cross-entropy because real-versus-fake classification has two classes. section 9.1.1 section 9.1.3
- Seq2Seq models take cross-entropy between predicted output tokens and target sequences as the learning signal. section 11.3
Binary GAN training
A GAN defines BCEWithLogitsLoss in PyTorch, labels real data with 1 and generated data with 0, and computes separate discriminator and generator losses. The discriminator learns from real and detached fake samples; the generator is trained to make fake samples receive the real label. section 9.1.3
