Decoder-only transformer
A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
Learn Generative AI with PyTorch
A Transformer design that uses only decoder stacks for autoregressive generation.
- How it is used
- Generates text from prompts and continues tokenized musical-event sequences without a separate input encoder.
- How deep it goes
- Explains next-token generation and specifies music-model embeddings, decoder layers, attention heads, and output logits.
- Examples
- Uses GPT-2 prompt completion and a six-layer MIDI-event music Transformer with a 390-token vocabulary.
- What you can do after
- Build it from scratch: implement GPT-2XL, load Hugging Face weights for prompted text generation, and construct a smaller music Transformer.
Read how Learn Generative AI with PyTorch explains decoder-only transformer →
Time Series Forecasting Using Foundation Models
A decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.
- How it is used
- Generates time-series forecasts autoregressively within probabilistic and deterministic foundation models.
- How deep it goes
- Explains patch tokens, sequential generation, self-attention, output patches, and expert routing, while stopping short of full implementation details.
- Examples
- Uses Lag-Llama, Moirai-MoE, and TimesFM to contrast probabilistic outputs, expert routing, and long-horizon output patches.
- What you can do after
- Evaluate decoder-only forecasting designs by comparing token granularity, autoregressive steps, output type, and architectural specialization.
Read how Time Series Forecasting Using Foundation Models explains decoder-only transformer →
Transformers in Action
A decoder-only architecture is a sequence-model architecture that generates outputs using a decoder without a separate input encoder.
- How it is used
- Supports autoregressive language generation, including text, code, and chat, rather than the representation-focused tasks associated with encoder-only models.
- How deep it goes
- Explains architectural removals, causal attention, next-token training, and token feedback, then illustrates preprocessing and attention with abstracted code.
- Examples
- Uses GPT-2 input tokenization and listings for masked multihead self-attention and a transformer block.
- What you can do after
- Evaluate/understand why a decoder-only model can generate tokens from prior context without encoder outputs.
Read how Transformers in Action explains decoder-only transformer →
Build a Text-to-Image Generator (from Scratch)
A decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.
- How it is used
- Uses autoregressive next-token prediction for prompt-based text generation and image-token generation.
- How deep it goes
- Explains layer structure, autoregressive flow, and EOS termination, but does not detail decoder-layer internals.
- Examples
- Uses ChatGPT for text generation, translation decoding with EOS, and min-DALL-E for image-token generation.
- What you can do after
- Evaluate/understand how decoder-only models turn prompts into sequences through repeated next-token prediction.
Read how Build a Text-to-Image Generator (from Scratch) explains decoder-only transformer →
Sutskever's List
A transformer language-model architecture that generates sequences using only a decoder stack, without a separate encoder.
- How it is used
- Uses decoder-only transformers to study how architecture choice and scale affect language-model performance.
- How deep it goes
- Explains left-to-right conditioning through masked self-attention and compares long-context performance and computational efficiency.
- Examples
- Contrasts LSTMs' short-sequence performance with their plateau and uses GPT-4's Tom Cruise example to illustrate directional brittleness.
- What you can do after
- Evaluate architecture trade-offs in long-context performance and computational efficiency at scale.
Read how Sutskever's List explains decoder-only transformer →
How Large Language Models Work
A decoder-only transformer is a transformer architecture composed of decoder-style layers that generates outputs autoregressively from preceding context.
- How it is used
- Generates text by continuing preceding context one token at a time.
- How deep it goes
- Explains embeddings, position, attention, probabilities, and sampling, but does not derive transformer mathematics or training procedures.
- Examples
- Uses GPT, Gemini, ChatGPT, and the continuation of a partially written document as examples.
- What you can do after
- Evaluate generated text by recognizing next-token prediction, probability-based output, and sampling randomness.
Read how How Large Language Models Work explains decoder-only transformer →
Effective Conversational AI
A decoder-only architecture is a language-model architecture trained to predict successive tokens and generate text from preceding context.
- How it is used
- Supports conversational generation when responses must address user requests with specificity or personalization.
- How deep it goes
- Defines next-token prediction, training style, and task range, but does not descend into transformer internals.
- Examples
- Uses conversational responses, classification, question answering, summarization, and next-token text completion as examples.
- What you can do after
- Evaluate whether generative modeling fits conversational tasks requiring responses beyond predefined intent categories.
Read how Effective Conversational AI explains decoder-only transformer →
Deep Learning with PyTorch, Second Edition
A transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.
- How it is used
- Generates contextually relevant text and supports the dominant architecture used by modern generative language applications.
- How deep it goes
- Builds a GPT-style decoder in PyTorch with embeddings, repeated blocks, attention projections, normalization, and output projection.
- Examples
- Demonstrates character-level name generation using a compact PyTorch model and displays generated sample names.
- What you can do after
- Build it from scratch in PyTorch and generate short text samples with a GPT-style decoder.
Read how Deep Learning with PyTorch, Second Edition explains decoder-only transformer →
Related concepts
- GPT modelincludes · A GPT model is a generative pretrained Transformer language model that produces text autoregressively by predicting successive tokens from preceding context.
- TimesFMused by · TimesFM is a foundation model designed for deterministic time-series forecasting.
- Moirai-MoEused by · Moirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.
- transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- encoder-only transformercontrasts with · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
- masked attentionuses · Masked attention is an attention mechanism in which a mask restricts which positions can influence each other, often preventing sequence positions from accessing future tokens.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.