liveBook · Manning

Sampling

As explained in Build an Advanced RAG Application (From Scratch)

A sampling technique generates text by probabilistically selecting tokens from a model's predicted distribution rather than always choosing the top-scoring token.

Sampling selects the next token probabilistically from an LLM's predicted distribution instead of always choosing the highest-scoring token. It introduces controlled randomness, trading some determinism for broader, more creative outputs. section 5.2.3 section 5.2

Three common controls shape the candidate distribution:

  • Temperature sampling divides log-probabilities by T: values above 1 increase randomness, values below 1 favor probable tokens, and values near 0 approach greedy decoding. section 5.2.3
  • Top-k sampling restricts selection to the k most probable tokens; k = 1 matches greedy decoding, while larger values increase diversity. section 5.2.3
  • Top-p, or nucleus, sampling selects from the smallest candidate subset whose cumulative probability reaches p, adapting the candidate set to the distribution. section 5.2.3

A short completion

Given “The quick brown fox,” top-k sampling with k = 3 considers only the three most likely continuations, while top-p sampling with p = 0.8 considers tokens covering 80% of the probability mass. Random selection among those candidates can produce alternatives such as “jumps,” “forest,” or “stream.” section 5.2.3

Formula: P(w_i) = exp(log(P(w_i)) / T) / sum_j(exp(log(P(w_j)) / T))

The temperature formula shows how dividing log-probabilities by T reshapes the next-token distribution before sampling. section 5.2.3

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines temperature, top-k, and top-p sampling with parameter effects, examples, advantages, and disadvantages, but stops short of implementation code.
Examples
Uses the prompt “The quick brown fox” with k = 3 and p = 0.8 to illustrate candidate restriction and probabilistic continuation.
What it assumes
Assumes predicted next-token probabilities and introduces the contrast between probabilistic sampling and deterministic decoding.
What you can do after
Apply it via temperature, top-k, and top-p controls to customize generated text for diversity, coherence, and user preferences.
How it connects
Connects sampling to decoding algorithms, greedy decoding, beam search, temperature scaling, top-k sampling, top-p sampling, and decoder models.

Also explained in

Compare how 31 books explain sampling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.