liveBook · Manning

Autoregressive model

As explained in Build a DeepSeek Model (From Scratch)

Autoregressive generation is a text-generation process that predicts one token at a time and feeds each generated token back into the context for the next prediction.

An autoregressive language model generates text one token at a time. After each prediction, it appends the new token to the sequence and uses the expanded sequence as context for the next prediction. Generation stops at an end-of-sequence token or a predefined token limit. section 2.1.2

Growing the context

Start with “The next day.” If the model predicts “is,” the next input becomes “The next day is.” If it then predicts “bright,” the context becomes “The next day is bright.” Each prediction depends on the prompt and the tokens already generated. section 2.1.2

A simple implementation passes the entire growing sequence through the model at every step. It selects the logits for the last position, chooses a token, and concatenates its ID to the input tensor. This makes the feedback loop explicit, but repeatedly processes earlier tokens. section 2.1.3 section 2.3

Listing 2.1 Visualizing autoregressive generation with GPT-2
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
 
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
 
prompt = "The next day is bright"
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs.input_ids
 
print(f"Prompt: '{prompt}'", end="")
 
for _ in range(20):
    outputs = model(input_ids)  #A
    logits = outputs.logits
 
    next_token_logits = logits[:, -1, :]  #B
    next_token_id = next_token_logits.argmax(dim=-1).unsqueeze(-1)
 
    input_ids = torch.cat([input_ids, next_token_id],
    [CA] dim=-1)  #C
 
    new_token = tokenizer.decode(next_token_id[0])
    print(new_token, end="", flush=True)
 
print("\n")
 

Inspect a PyTorch loop that selects the highest-logit token from pretrained GPT-2 and appends it to the input. section 2.1.3

The redundancy is visible inside attention. When a four-token input grows to five tokens, the earlier query, key, and value projections remain unchanged with fixed inference weights. The top-left 4×4 block of the new attention-score matrix repeats the previous calculation, even though only the last position's context vector is needed to predict the next token. section 2.3.2 section 2.3

A key-value (KV) cache avoids recomputing past key and value projections:

  • Prefill: process the prompt in one forward pass and populate the cache. section 2.4.2
  • Decode: compute query, key, and value vectors for the new token; append its key and value to the cache; then use the new query with the updated cache to calculate attention and predict the next token. section 2.4.2
  • Keep caches per layer and attention head. Reuse reduces computation, but cache memory grows with sequence length, layers, and heads. section 2.4.2 section 2.9

For attention, caching reduces the stated cost of each new-token step from quadratic to linear in sequence length, and total generation from cubic to quadratic. Multi-query and grouped-query attention reduce cache memory by sharing key and value projections, with a trade-off in model expressivity. section 2.9

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces generation through a concrete loop, repeated attention matrices, and a per-layer, per-head caching workflow.
Examples
Traces “The next day” through successive predictions and demonstrates token generation with pretrained GPT-2.
What you can do after
Apply it via PyTorch and Transformers to generate tokens with pretrained GPT-2.
How it connects
Connects next-token prediction to causal attention, KV caching, multi-query attention, and grouped-query attention.

Also explained in

Compare how 8 books explain autoregressive model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.