Top-k sampling is a decoding strategy that restricts next-token selection to the k most probable candidates, then samples randomly among them. This preserves high-probability continuations while adding controlled diversity to autoregressive generation. section 5.2.3 section 5.2
Configure top-k sampling by choosing the candidate-set size:
- Set k = 1 to reproduce greedy decoding, selecting only the most probable next token. section 5.2.3
- Increase k to consider more high-probability alternatives and produce more diverse output. section 5.2.3
- Treat small k values as a quality-diversity trade-off: they are easy to control but may exclude less probable, relevant tokens. section 5.2.3
- Contrast top-k with top-p sampling, which varies the candidate set by cumulative probability rather than a fixed token count. section 5.2.3
Fox continuation with k = 3
Given the prompt “The quick brown fox,” top-k sampling with k = 3 keeps only the three most likely next words. The next token is then chosen randomly according to their probabilities, allowing continuations such as “jumps,” “over,” or another high-probability option. section 5.2.3
Figure 5.8 Top-k sampling in LLMs, where the model generates a probability distribution for the next word, and only the top-k most probable words are considered, with the final word chosen randomly based on their probabilities.
Use the figure to inspect how a next-token probability distribution is reduced to its top-k candidates before random selection. section 5.2.3








