Cross-entropy loss measures the difference between predicted class probabilities and the true labels. It is commonly used when a model chooses among multiple categories, including clothing classes or the next token in a sequence. section 2.2.1 section 14.1.3
For multicategory classification, PyTorch’s nn.CrossEntropyLoss() accepts raw output logits: it combines LogSoftmax and NLLLoss internally, so applying softmax before the loss is unnecessary. Softmax converts the class scores into values that sum to one; binary classification instead uses a sigmoid-style output. section 2.4.2
Training a ten-class clothing classifier
A dense network produces ten logits for ten clothing categories. The training loop computes loss = loss_fn(preds, labels), clears gradients, backpropagates the loss, and updates weights with Adam. Validation repeats the loss calculation without updating the model. The resulting class predictions can be compared with labels to calculate accuracy.
Predicting the next music token
A music Transformer compares each predicted token distribution with the shifted target sequence. Padding can be excluded with ignore_index=389, while gradient clipping limits the gradient norm during optimization. section 14.1.3 section 14.4.1
