One-hot encoding represents one categorical outcome with a vector of m elements. Exactly one element is 1, identifying the observed class; every other element is 0. For a four-class image classifier, a cat label can be written as [1, 0, 0, 0], while a dog label is [0, 1, 0, 0]. section 5.9.6 section 9.1.3
The vector is also a compact representation of a categorical-distribution outcome. If the class probabilities are π₁, …, πₘ and the observed class is i, the one-hot vector has xᵢ = 1 and xⱼ = 0 for j ≠ i. In categorical probability expressions, the zero-valued powers disappear, leaving the probability πᵢ of the observed class. section 5.9.6
One-hot targets connect directly to classifier training:
- Compare the one-hot ground truth with a prediction vector whose entries represent class probabilities. section 6.3
- Compute cross-entropy as −Σᵢ yᵢ log(pᵢ). Because only the correct-label entry yᵢ equals 1, the expression reduces to −log(pcorrect). section 9.1.3
- Use a confident correct prediction such as [0.003, 0.997, 0, 0] for a dog target [0, 1, 0, 0]; its loss is low. section 9.1.6
PyTorch cross-entropy
For prediction probabilities [0.8, 0.15, 0.04, 0.01] and cat target [1, 0, 0, 0], compute loss with `-torch.dot(y_gt, torch.log(y_pred))`. For logits, PyTorch can combine softmax and cross-entropy with `cross_entropy(scores, y_gt)`; the target may be supplied as the class index `1`. section 9.1.3 section 9.1.6
Listing 9.5 PyTorch code for softmax cross-entropy loss
from torch.nn.functional import cross_entropy scores = torch.tensor([[9.99, 10, 0.01, -10]]) y_gt = torch.tensor([1]) #1 loss = cross_entropy(scores, y_gt) #2
Inspect the PyTorch softmax cross-entropy listing to connect class scores, the target class, and the resulting loss. section 9.1.6
