Autoregressive generation
Autoregressive generation produces a sequence one element at a time, conditioning each new element on the elements already available.
CUDA for LLMs
Autoregressive generation produces a sequence one token at a time, conditioning each new token on preceding tokens.
- How it is used
- Uses autoregressive generation as the sequential decoding workload for transformer inference optimization.
- How deep it goes
- Traces sampling, cache reuse, decode tensor shapes, and CUDA-backed inference, including the memory–computation tradeoff.
- Examples
- Uses a prompt continuation and a 100-token cache accounting example with a 24-layer transformer configuration.
- What you can do after
- Evaluate and understand redundant decoding work and the memory tradeoff of KV caching.
Timeless Algorithms, The Foundational Ideas
Autoregressive generation is the process of producing a sequence by predicting one token at a time, conditioning each prediction on tokens already available.
- How it is used
- Explains how successive next-token predictions build a multistep response.
- How deep it goes
- Works through causal masking and a multistep example, using simplified operations rather than reproducing a complete trained transformer.
- Examples
- Uses an overtaking-trains calculation and a four-token attention mask to illustrate chained predictions and available context.
- What you can do after
- Evaluate and understand how sequential dependencies affect generation, latency, and reliability.
Read how Timeless Algorithms, The Foundational Ideas explains autoregressive generation →
Rearchitecting LLMs
Autoregressive generation produces a sequence one token at a time, conditioning each new token on previously generated tokens.
- How it is used
- Explains how depth pruning affects inference memory and why static benchmarks may miss generation degradation.
- How deep it goes
- Defines token-by-token dependence and explains per-layer key/value storage as the cache grows with sequence length.
- Examples
- Contrasts one-token benchmark answers with short-prompt coherence and long-form factual inconsistencies after layer removal.
- What you can do after
- Evaluate whether pruned models retain coherent text generation instead of relying only on static benchmarks.
Read how Rearchitecting LLMs explains autoregressive generation →
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- attention headAn attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
- inner productAn inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- feedforward networkA feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
Related concepts
- tokenproduces · A token is a value used to determine a data partition's placement and ownership in a distributed database cluster.
- key-value cacheuses · A key-value cache stores attention keys and values computed for earlier tokens so they can be reused during subsequent autoregressive generation.
- decode phasehas part · The decode phase is the autoregressive inference stage in which a model generates output tokens sequentially from previously processed context.
- prefillhas part · Prefill is the initial inference phase in which a model processes the available input context and prepares state for subsequent autoregressive generation.
- inferenceused by · Inference is the process of applying a trained model to input data to produce predictions or other outputs.
- latencyproduces · Latency is the elapsed time between initiating an operation or request and receiving its corresponding result or observable response.
Questions about autoregressive generation
What is autoregressive generation?
Autoregressive generation produces a sequence one element at a time, conditioning each new element on the elements already available.
Which Manning books explain autoregressive generation?
CUDA for LLMs, Timeless Algorithms, The Foundational Ideas and Rearchitecting LLMs.
What should I know before learning autoregressive generation?
Manning books that teach autoregressive generation usually introduce attention, attention head, inner product and feedforward network first.
What are the parts of autoregressive generation?
Autoregressive generation includes decode phase and prefill.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


