Top-p filtering, also called nucleus sampling, generates each next token from the smallest probability-ranked subset whose cumulative probability reaches a threshold p. It replaces a fixed candidate count with an adaptive probability mass, balancing coherence and diversity. section 5.0 section 5.2.3
At each generation step, top-p filtering:
- Ranks possible next tokens by probability. section 5.2.3
- Accumulates probability mass until it reaches the selected threshold p. section 5.2.3
- Samples randomly from that nucleus, allowing several plausible continuations rather than only the most likely token. section 5.2.3
A nucleus after “The quick brown fox”
With p = 0.8, retain the highest-probability words until they collectively cover 80% of the probability mass. The next token may then vary among plausible continuations such as “jumps,” “forest,” or “stream,” rather than being restricted to a fixed number of candidates. section 5.2.3
Compared with top-k sampling, top-p filtering adapts the candidate-set size to the distribution and offers flexible randomness control, but may cost more computation for large vocabularies. Its behavior can also interact with temperature settings. section 5.2.3 section 5.2













