A language model assigns probabilities to possible next tokens; a decoding strategy determines how one is selected. Selection is distinct from scoring: the same distribution can yield a predictable choice or allow variation. The chosen token joins the context for the next prediction. section 12.2.1
Two contrasting strategies illustrate the choice:
- Greedy decoding selects the highest-probability token. section 12.2.1
- Sampling selects among candidates according to their probabilities, so a lower-probability token may be chosen. section 12.2.1
A pronoun-resolution continuation
For “The trophy didn’t fit in the box because it was too…,” the illustrative distribution assigns “small” probability 0.48 and “large” probability 0.25. Greedy decoding chooses “small”; sampling can sometimes choose “large.” section 12.2.1













