liveBook · Manning

Autoregressive model

As explained in Build an Advanced RAG Application (From Scratch)

An autoregressive model generates each successive output element conditioned on previously generated elements.

An autoregressive model generates a sequence one token at a time, predicting each next token from the prompt and all previously generated tokens. This causal chain maintains context and supports coherent, contextually appropriate output. section 5.1.1

P(S) = P(w 1) × P(w 2|w 1) × ... × P(w n|w 1, w 2, ..., w n-1)

The sequence probability is decomposed into conditional next-token probabilities, making the model's stepwise generation explicit. section 5.1.1

Autoregressive generation supports:

  • Maintaining context across long passages while applying patterns learned during training. section 5.1.1
  • Producing content, translations, and conversations through decoder models. section 5.7
  • Using decoding algorithms such as greedy decoding, beam search, and sampling to balance coherence, diversity, and computational efficiency. section 5.7

Next-token completion

Given “The cat sat on the...”, the model evaluates possible next words and may select “mat” over “cloud” or “computer” because the preceding context makes it more appropriate. section 5.1.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains token-level conditioning, sequence-probability factorization, decoder processing, and decoding choices without detailing training internals.
Examples
Uses next-word completion, decoder-only Transformer processing, and a hotel-search system that analyzes retrieved results.
What you can do after
Evaluate/understand how decoder models generate context-sensitive sequences and how decoding choices affect their outputs.
How it connects
Connects autoregressive generation with decoder models, attention mechanisms, decoding algorithms, prompting techniques, and retrieval-augmented generation.

Also explained in

Compare how 8 books explain autoregressive model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.