A decoding strategy structures how a model generates an output sequence from an input sequence. It navigates a large space of possible token sequences by determining how the next token is selected, affecting coherence, creativity, and diversity. section 4.1 section 4.3
Common strategies differ in search breadth, determinism, and output diversity:
- Greedy search selects the most probable token at every step, making it efficient but unable to reconsider earlier choices. section 4.1.1
- Beam search maintains multiple candidate sequences, often improving fluency while tending toward similar outputs. section 4.3
- Top-k and nucleus sampling restrict stochastic selection to high-probability tokens, promoting diversity. section 4.3
- Temperature sampling sharpens or flattens the probability distribution to control randomness; combining it with top-p balances coherence and novelty. section 4.3
Greedy generation with model.generate
Tokenize a prompted completion request, load a causal language model, and call model.generate with do_sample=False and max_new_tokens=128. This produces a deterministic continuation by selecting the most probable next token at each step. section 4.1.1
