liveBook · Manning

Decoding strategy

As explained in Build an Advanced RAG Application (From Scratch)

A decoding algorithm is a procedure for selecting or generating output tokens from a language model's predicted probability distribution.

A decoding strategy converts a language model’s predicted probability distribution for the next token into generated text. It controls the balance among coherence, diversity, style, and computational efficiency. section 5.2 section 5.7

Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.

Autoregressive generation processes the preceding token sequence through decoder components and produces a probability distribution for the next token; decoding then selects from that distribution. section 5.1

Common decoding strategies include:

  • Greedy decoding, which selects a next-token output directly from the model’s probabilities. section 5.2
  • Beam search, which provides an alternative decoding approach for generating text. section 5.2
  • Sampling methods, including top-k sampling, top-p (nucleus) sampling, and temperature scaling, which adjust how outputs are selected. section 5.2

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines the selection problem and names major algorithms, but stops short of their internal procedures or parameter settings.
Examples
Names greedy decoding, beam search, top-k sampling, top-p sampling, and temperature scaling as concrete strategies.
What it assumes
Assumes familiarity with decoder models and introduces their next-token probability prediction as the basis for selection.
What you can do after
Apply named decoding algorithms to balance diversity, coherence, and computational efficiency for specific generation use cases.
How it connects
Connects decoder models, autoregressive language models, token probabilities, prompting techniques, and Retrieval Augmented Generation.

Also explained in

Compare how 14 books explain decoding strategy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.