A decoding strategy converts a language model’s predicted probability distribution for the next token into generated text. It controls the balance among coherence, diversity, style, and computational efficiency. section 5.2 section 5.7
Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.
Autoregressive generation processes the preceding token sequence through decoder components and produces a probability distribution for the next token; decoding then selects from that distribution. section 5.1
Common decoding strategies include:
- Greedy decoding, which selects a next-token output directly from the model’s probabilities. section 5.2
- Beam search, which provides an alternative decoding approach for generating text. section 5.2
- Sampling methods, including top-k sampling, top-p (nucleus) sampling, and temperature scaling, which adjust how outputs are selected. section 5.2













