liveBook · Manning

Decoding strategy

As explained in Timeless Algorithms, The Foundational Ideas

A procedure for selecting a generated token from a model’s next-token probability distribution.

A language model assigns probabilities to possible next tokens; a decoding strategy determines how one is selected. Selection is distinct from scoring: the same distribution can yield a predictable choice or allow variation. The chosen token joins the context for the next prediction. section 12.2.1

Two contrasting strategies illustrate the choice:

  • Greedy decoding selects the highest-probability token. section 12.2.1
  • Sampling selects among candidates according to their probabilities, so a lower-probability token may be chosen. section 12.2.1

A pronoun-resolution continuation

For “The trophy didn’t fit in the box because it was too…,” the illustrative distribution assigns “small” probability 0.48 and “large” probability 0.25. Greedy decoding chooses “small”; sampling can sometimes choose “large.” section 12.2.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines token selection and contrasts greedy decoding with sampling, without detailing other methods.
Examples
Compares “small” and “large” as continuations of a pronoun-resolution sentence.
What you can do after
Apply greedy decoding or sampling to select a token from an illustrative distribution.
How it connects
Connects greedy decoding and sampling to probabilistic language-model generation.

Also explained in

Compare how 14 books explain decoding strategy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.