Top-p filtering, or nucleus sampling, samples each next token from the smallest post-softmax candidate set whose cumulative probability exceeds threshold p. Unlike top-k sampling, its candidate-set size changes with model confidence: confident predictions narrow the set, while uncertain predictions broaden it. section 4.1.4
Use top-p to tune the trade-off between predictability and variety:
- Set p between 0.8 and 0.95 to control how broadly the sampler explores high-probability tokens. section 4.1.4
- Combine top-p with temperature to balance coherence and novelty for a particular generation task. section 4.3
Hugging Face generation
Enable nucleus sampling with model.generate(..., do_sample=True, top_p=0.9, temperature=1.0, num_return_sequences=3). A threshold of 0.9 can restrict the first choice to tokens such as “movie” or “pie,” then dynamically select later candidates such as “was” or “is.” section 4.1.4
Figure 4.3 If we assume we want to model the probability of the next word to be chosen with top-p (nucleus) sampling, we would again consider the following: P(w |mid “The”), which will then be followed the probability P(w |mid “The,” “movie”), and so on. But now the choice of the model will be limited to the threshold of cumulative probability of 0.9 for the words which can be selected, so again, only movie or pie will be considered for the first probability and was or is for the next choice, respectively.
Visualizes cumulative probability at 0.9 across successive next-token choices. section 4.1.4
