Top-p filtering, also called nucleus sampling, limits stochastic decoding to the smallest high-probability token set whose cumulative probability reaches a threshold p. It reduces accidental sampling of unrelated, low-confidence tokens while preserving more diversity than greedy decoding. section 4.5 section 4.7
Apply the filter after temperature scaling and softmax, but before sampling:
- Sort token probabilities in descending order and compute their cumulative sum. section 4.5.1
- Keep the prefix whose cumulative mass is below p; the common variant uses prefix mass before each token, thereby retaining the token that first crosses p. section 4.5.1 section 4.5.2
- Zero out discarded probabilities, scatter retained values back to their original vocabulary positions, and renormalize them to sum to one. section 4.5 section 4.5.2
- Sample from the filtered distribution with torch.multinomial; bypass filtering when top_p is unset or at least 1.0. section 4.5.2
A p = 0.8 cutoff
For a ten-token toy vocabulary, sorted probabilities accumulate as 0.4538, 0.7290, and 0.8119. The simple rule keeps two tokens; the common prefix rule also keeps the third token that crosses the threshold. After filtering, the retained values are renormalized, producing probabilities such as 0.5589, 0.1021, and 0.3390 in the original vocabulary order. With real next-token logits and temperature 0.35, applying top_p=0.8 removes rare outputs such as “Munich” and “Hamburg” while retaining likely city or formatting tokens. section 4.5.1 section 4.5.2
Figure 4.12 Overview of the top-p filtering process. The filter keeps only the highest-probability tokens by sorting them, applying a cumulative cutoff, selecting the top-p subset, and renormalizing the result.
Use the workflow figure to inspect sorting, cumulative cutoff, subset selection, and renormalization as one decoding pipeline. section 4.5
