Sampling selects the next token probabilistically from the distribution conditioned on tokens already generated. Unlike greedy decoding, it may choose the second- or third-most-probable token, so repeated inference runs can produce different sentences. section 10.2
Temperature controls how often decoding moves farther down the probability ranking: higher variation can produce creative but nonsensical text, while constrained choices reduce unpredictability. Sampling complements greedy search, repetition filtering, and beam search as a token-decoding strategy. section 10.2 section 10.2.1
Nucleus sampling with GPT-2
A runnable Hugging Face workflow loads GPT-2, tokenizes a prompt, generates text, and decodes the result. Setting do_sample=True and top_p=0.92 enables nucleus sampling, which restricts choices to a probability nucleus. For the prompt “NLP is a”, one output continues with an improvised explanation of network protocols, illustrating both fluent structure and factual unreliability. section 10.2.1
Figure 10.2 Stochastic chameleons deciding words one at a time
A fishbone-style decision tree makes stochastic token selection visible by showing ranked alternatives and the chosen word at each generation step. section 10.2
