A logit is a raw numerical score produced by a model before scores are normalized into probabilities by softmax. Logits may have any real value. section 13.4.2 section 7.2.4
Logits play different roles during training and prediction:
- Feed logits directly to cross-entropy loss; the loss internally combines softmax with negative log likelihood and offers numerical-stability benefits. section 13.4.2 section 7.2.4
- Convert logits to probabilities with softmax when class probabilities are needed for classification. section 13.4.2
- Avoid treating logits as probabilities: they are not restricted to the range 0 to 1 and do not need to sum to 1. section 13.4.2 section 7.2.4
From scores to probabilities
For logits [1, -2, 3], exponentiation gives approximately [2.718, 0.135, 20.086]. Dividing by their sum produces softmax probabilities [0.118, 0.006, 0.876], so the third class is selected. section 13.4.2
Listing 13.6 LunaModel forward pass
def forward(self, input_batch):
bn_output = self.tail_batchnorm(input_batch)
block_out = self.block1(bn_output)
block_out = self.block2(block_out)
block_out = self.block3(block_out)
block_out = self.block4(block_out)
conv_flat = block_out.view(
block_out.size(0), #1
-1,
)
linear_output = self.head_linear(conv_flat)
return linear_output, self.head_softmax(linear_output)
A classification model returns its linear-layer output as logits and applies softmax separately to obtain probabilities. section 13.4.2
