Sampling selects the next token probabilistically from an LLM's predicted distribution instead of always choosing the highest-scoring token. It introduces controlled randomness, trading some determinism for broader, more creative outputs. section 5.2.3 section 5.2
Three common controls shape the candidate distribution:
- Temperature sampling divides log-probabilities by T: values above 1 increase randomness, values below 1 favor probable tokens, and values near 0 approach greedy decoding. section 5.2.3
- Top-k sampling restricts selection to the k most probable tokens; k = 1 matches greedy decoding, while larger values increase diversity. section 5.2.3
- Top-p, or nucleus, sampling selects from the smallest candidate subset whose cumulative probability reaches p, adapting the candidate set to the distribution. section 5.2.3
A short completion
Given “The quick brown fox,” top-k sampling with k = 3 considers only the three most likely continuations, while top-p sampling with p = 0.8 considers tokens covering 80% of the probability mass. Random selection among those candidates can produce alternatives such as “jumps,” “forest,” or “stream.” section 5.2.3
The temperature formula shows how dividing log-probabilities by T reshapes the next-token distribution before sampling. section 5.2.3






























