Sampling is how a language model turns its output probabilities into one next token. For every vocabulary token, it estimates a probability, then randomly selects one according to those probabilities. The selected token joins the input, and the model repeats the process to generate text. section 3.2.3 section 3.5
Figure 3.10 Producing output from LLMs involves converting from documents to tokens and then using the model to produce output. We loop through this process to both consume text and generate human-readable output.
Follow the loop from tokenized input through model output and back into the next generation step. section 3.2.3
Why randomness helps
Given “I love to eat,” foods such as barbeque and sushi receive high probabilities, while a car and 42 receive low probabilities. Weighted random selection might choose “tacos” rather than always choosing the single most likely food. Generation continues until the model produces an end-of-sequence token or reaches a maximum limit. section 3.2.3
This randomness explains why the same prompt can produce different answers. Increasing or reducing its strength trades consistency against creativity and variety; randomness can improve natural-sounding text but does not guarantee accuracy. section 3.2.3 section 3.5
