Top-p filtering controls how much of the next-token probability distribution remains available for sampling. It keeps the most likely tokens until their cumulative probability reaches a chosen threshold, adapting the candidate-set size to the distribution. section 7.2.2 section 9.1.2
A threshold changes the candidate set
Suppose the candidate tokens “woman” and “man” have probabilities of 0.35 and 0.26. With top-p set to 0.5, both remain because their combined probability exceeds 0.5; lower-probability tokens are excluded. A very low threshold such as 0.05 produces rigid responses, while 0.95 permits more flexibility but can increase gibberish. section 7.2.2
Top-p is useful when generation needs a practical balance between control and variation:
- Unlike top-k filtering, it limits the cumulative probability mass rather than a fixed number of tokens. section 7.2.2 section 9.1.2
- Generation first obtains logits for the final input position, applies temperature, and can then sample using top_p alongside top_k. section 9.1.2
- Use multinomial sampling to explore alternatives; compare it with argmax and compare top-p with top-k to observe output differences. section 9.1.2
Figure 7.3 A simple path of how the next word is chosen. Given an input, a model will generate a vector of logits for each token in the model’s vocabulary. Using the softmax algorithm, these logits will be transformed into probabilities. These probabilities will correspond to how often that token is likely to be chosen. Temperature is applied during the softmax algorithm.
Trace the path from model logits through softmax probabilities to next-token selection, including where temperature affects the distribution. section 7.2.2
