Softmax converts a vector of scores into nonnegative probabilities that sum to one. In attention, it turns query–key compatibility scores into weights for combining value vectors. In distillation, it turns teacher logits into a probability distribution over vocabulary tokens. section 3.1.2 section 6.3.1
Scores and log probabilities
For logits [2.0, 1.0, 0.1], softmax produces approximately [0.659, 0.242, 0.099]. The corresponding log_softmax values are approximately [-0.417, -1.417, -2.317]: natural logarithms of the probabilities. section 6.3.1
In attention, a token’s query is compared with each token’s key using dot products. Softmax normalizes the resulting scores; the weights determine how much of each value contributes to the token’s enriched embedding. section 3.1.2
For distillation with PyTorch’s F.kl_div, use log_softmax for student predictions and softmax for teacher predictions. The first argument must be in log space, which helps avoid numerical problems with very small probabilities. Computing a logarithm after softmax can produce invalid values in extreme cases. section 6.3.1
student_soft = F.log_softmax(
student_logits / temperature,
dim=-1) #A
teacher_soft = F.softmax(
teacher_logits / temperature,
dim=-1) #B
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
loss_logits = F.kl_div(
student_soft.view(-1, student_soft.size(-1)),
teacher_soft.view(-1, teacher_soft.size(-1)),
reduction='batchmean' #C
) * (temperature ** 2) #D
This workflow divides both sets of logits by temperature, normalizes over the last dimension, removes the final sequence position, and flattens the remaining positions for KL divergence. The loss uses batchmean reduction and is multiplied by temperature squared. section 6.3.1

























