liveBook · Manning

Softmax

As explained in Rearchitecting LLMs

Softmax converts a vector of scores into nonnegative probabilities that sum to one.

Softmax converts a vector of scores into nonnegative probabilities that sum to one. In attention, it turns query–key compatibility scores into weights for combining value vectors. In distillation, it turns teacher logits into a probability distribution over vocabulary tokens. section 3.1.2 section 6.3.1

Scores and log probabilities

For logits [2.0, 1.0, 0.1], softmax produces approximately [0.659, 0.242, 0.099]. The corresponding log_softmax values are approximately [-0.417, -1.417, -2.317]: natural logarithms of the probabilities. section 6.3.1

In attention, a token’s query is compared with each token’s key using dot products. Softmax normalizes the resulting scores; the weights determine how much of each value contributes to the token’s enriched embedding. section 3.1.2

For distillation with PyTorch’s F.kl_div, use log_softmax for student predictions and softmax for teacher predictions. The first argument must be in log space, which helps avoid numerical problems with very small probabilities. Computing a logarithm after softmax can produce invalid values in extreme cases. section 6.3.1

student_soft = F.log_softmax(
    student_logits / temperature, 
    dim=-1) #A
teacher_soft = F.softmax(
    teacher_logits / temperature, 
    dim=-1) #B
 
student_soft = student_soft[..., :-1, :].contiguous()
teacher_soft = teacher_soft[..., :-1, :].contiguous()
 
loss_logits = F.kl_div(
    student_soft.view(-1, student_soft.size(-1)),
    teacher_soft.view(-1, teacher_soft.size(-1)),
    reduction='batchmean' #C
) * (temperature ** 2) #D

This workflow divides both sets of logits by temperature, normalizes over the last dimension, removes the final sequence position, and flattens the remaining positions for KL divergence. The loss uses batchmean reduction and is multiplied by temperature squared. section 6.3.1

How Rearchitecting LLMs covers it

How deep it goes
Explains normalization and a PyTorch distillation workflow but does not derive the softmax formula.
Examples
Uses logits [2.0, 1.0, 0.1] and an attention query for “spring” to illustrate normalization.
What you can do after
Apply it via PyTorch alongside log_softmax and KL divergence to compute a distillation logits loss.
How it connects
Connects softmax to attention, query–key dot products, knowledge distillation, log_softmax, and KL divergence.

Also explained in

Compare how 26 books explain softmax →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.