liveBook · Manning

Top-k sampling

As explained in Build an Advanced RAG Application (From Scratch)

Top-k sampling restricts next-token selection to the k highest-probability candidate tokens before sampling among them.

Top-k sampling is a decoding strategy that restricts next-token selection to the k most probable candidates, then samples randomly among them. This preserves high-probability continuations while adding controlled diversity to autoregressive generation. section 5.2.3 section 5.2

Configure top-k sampling by choosing the candidate-set size:

  • Set k = 1 to reproduce greedy decoding, selecting only the most probable next token. section 5.2.3
  • Increase k to consider more high-probability alternatives and produce more diverse output. section 5.2.3
  • Treat small k values as a quality-diversity trade-off: they are easy to control but may exclude less probable, relevant tokens. section 5.2.3
  • Contrast top-k with top-p sampling, which varies the candidate set by cumulative probability rather than a fixed token count. section 5.2.3

Fox continuation with k = 3

Given the prompt “The quick brown fox,” top-k sampling with k = 3 keeps only the three most likely next words. The next token is then chosen randomly according to their probabilities, allowing continuations such as “jumps,” “over,” or another high-probability option. section 5.2.3

Figure 5.8 Top-k sampling in LLMs, where the model generates a probability distribution for the next word, and only the top-k most probable words are considered, with the final word chosen randomly based on their probabilities.

Use the figure to inspect how a next-token probability distribution is reduced to its top-k candidates before random selection. section 5.2.3

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines fixed-candidate filtering and random selection, illustrates k values, and stops short of implementation code.
Examples
Uses k = 1 and k > 1 comparisons plus a k = 3 continuation of “The quick brown fox.”
What it assumes
Assumes familiarity with next-token probabilities and introduces top-k within the broader family of decoding algorithms.
What you can do after
Apply it by choosing a candidate-set size and sampling among high-probability next-token alternatives.
How it connects
Connects top-k sampling with autoregressive language models, greedy decoding, top-p sampling, temperature scaling, and decoding algorithms.

Also explained in

Compare how 9 books explain top-k sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.