A logit is an unnormalized real-valued score assigned to each candidate token before probability normalization. For a prompt such as “I am a,” the model produces one logit vector across its vocabulary. section 7.2.2
Generation reads the logits from the final input position, adjusts them with temperature, and converts them with softmax into probabilities. Higher temperature flattens the distribution; lower temperature favors tokens with larger logits. section 9.1.2 section 7.2.2
Logits support several next-token selection strategies:
- Use argmax to select the token with the highest logit, producing deterministic output. section 7.2.2
- Use multinomial sampling for more exploration after converting logits into probabilities. section 9.1.2
- Apply top-k or top-p filtering to restrict sampling to the most probable candidates. section 9.1.2
Reading the final-step logits
A generation loop obtains model output, keeps logits for the last time step, divides them by temperature, optionally applies top-k filtering, and passes the result to softmax. section 9.1.2
Figure 7.3 A simple path of how the next word is chosen. Given an input, a model will generate a vector of logits for each token in the model’s vocabulary. Using the softmax algorithm, these logits will be transformed into probabilities. These probabilities will correspond to how often that token is likely to be chosen. Temperature is applied during the softmax algorithm.
Trace the path from a prompt through vocabulary logits and softmax probabilities to the selected next token. section 7.2.2
