liveBook · Manning

Nucleus sampling

As explained in Build an Advanced RAG Application (From Scratch)

Top-p, or nucleus, sampling selects from the smallest probability-ranked token set whose cumulative probability reaches a specified threshold.

Top-p filtering, also called nucleus sampling, generates each next token from the smallest probability-ranked subset whose cumulative probability reaches a threshold p. It replaces a fixed candidate count with an adaptive probability mass, balancing coherence and diversity. section 5.0 section 5.2.3

At each generation step, top-p filtering:

  • Ranks possible next tokens by probability. section 5.2.3
  • Accumulates probability mass until it reaches the selected threshold p. section 5.2.3
  • Samples randomly from that nucleus, allowing several plausible continuations rather than only the most likely token. section 5.2.3

A nucleus after “The quick brown fox”

With p = 0.8, retain the highest-probability words until they collectively cover 80% of the probability mass. The next token may then vary among plausible continuations such as “jumps,” “forest,” or “stream,” rather than being restricted to a fixed number of candidates. section 5.2.3

Compared with top-k sampling, top-p filtering adapts the candidate-set size to the distribution and offers flexible randomness control, but may cost more computation for large vocabularies. Its behavior can also interact with temperature settings. section 5.2.3 section 5.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines cumulative-probability selection, contrasts top-k trade-offs, and illustrates one toy continuation without implementation code.
Examples
Uses p = 0.8 with “The quick brown fox” and visualizes probability-mass filtering in Figure 5.9.
What it assumes
Assumes familiarity with next-token probabilities and introduces top-p alongside other decoding methods.
What you can do after
Apply it via a decoder to adjust candidate-set flexibility and randomness when generating diverse, coherent text.
How it connects
Connects autoregressive language models with decoding algorithms, top-k sampling, temperature scaling, greedy decoding, and beam search.

Also explained in

Compare how 14 books explain nucleus sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.