Classification accuracy measures the fraction or percentage of predictions that match the correct class labels. For spam detection, the model predicts either 0 for non-spam or 1 for spam. Because classification uses two output values rather than the vocabulary-sized output used in pretraining, prediction takes the final-token logits and selects the largest value with argmax. section 6.6 section 6.9
Calculate accuracy across batches by following this flow:
- Run the model in evaluation mode and read model(input_batch)[:, -1, :], producing one two-class logit vector per input. section 6.6
- Convert each vector to a predicted label with torch.argmax(logits, dim=-1); applying softmax first is optional because it does not change the largest index. section 6.6
- Count predictions equal to target_batch, divide correct predictions by the number of examples, and optionally limit evaluation to a fixed number of batches. section 6.6
Before and after fine-tuning
For logits [-3.5983, 3.9902], argmax returns class 1, meaning spam. Before training, sample accuracies are near the 50% random baseline: 46.25% training, 45.00% validation, and 48.75% test. After fine-tuning, they reach 97.21%, 97.32%, and 95.67%. section 6.6 section 6.7
Figure 6.14 The model outputs corresponding to the last token are converted into probability scores for each input text. The class labels are obtained by looking up the index position of the highest probability score. The model predicts the spam labels incorrectly because it has not yet been trained.
Trace how final-token logits become class probabilities and labels, including why an untrained model can misclassify spam examples. section 6.6
