Decoder-only transformer
A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.
Learn Generative AI with PyTorch
A Transformer design that uses only decoder stacks for autoregressive generation.
- How it is used
- Uses decoder-only transformers for autoregressive text and music generation from preceding tokens.
- How deep it goes
- Explains generation and decoder-layer architecture, then specifies embeddings, attention heads, vocabulary, and sequence length for a music model.
- Examples
- Uses GPT-2XL prompt completion and a six-layer music transformer operating on MIDI event tokens.
- What you can do after
- Builds decoder-only models from scratch and loads pretrained GPT-2 weights to generate text.
Read how Learn Generative AI with PyTorch explains decoder-only transformer →
CUDA for LLMs
A decoder-only transformer is a transformer architecture that generates sequences using causal attention over preceding tokens.
- How it is used
- Uses decoder-only transformers as the baseline workload for character-level modeling and CUDA inference.
- How deep it goes
- Traces the model from embeddings through masked attention, residual blocks, logits, and loss, then introduces inference phases and caching.
- Examples
- Uses a 65-character vocabulary and an eight-block model trained on The Wonderful Wizard of Oz.
- What you can do after
- Build a character-level transformer forward pass in PyTorch with masked attention and vocabulary logits.
Fight Fraud with Machine Learning
A decoder-only architecture is a generative sequence-modeling design containing decoder blocks without a separate encoder component.
- How it is used
- Uses decoder blocks to generate prompt-conditioned text one token at a time.
- How deep it goes
- Explains causal attention, iterative next-token prediction, removed encoder-decoder attention, and byte-pair encoding without decoder-block implementation details.
- Examples
- Illustrates next-word prediction, greedy search, English-to-French translation, and GPT autoregressive generation diagrams.
- What you can do after
- Evaluate/understand why GPT generates sequentially and differs structurally from an encoder-decoder translation model.
Read how Fight Fraud with Machine Learning explains decoder-only transformer →
Time Series Forecasting Using Foundation Models
A decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.
- How it is used
- Generates time-series forecasts autoregressively within probabilistic and deterministic foundation models.
- How deep it goes
- Explains patch tokens, sequential generation, self-attention, output patches, and expert routing, while stopping short of full implementation details.
- Examples
- Uses Lag-Llama, Moirai-MoE, and TimesFM to contrast probabilistic outputs, expert routing, and long-horizon output patches.
- What you can do after
- Evaluate decoder-only forecasting designs by comparing token granularity, autoregressive steps, output type, and architectural specialization.
Read how Time Series Forecasting Using Foundation Models explains decoder-only transformer →
Transformers in Action
A decoder-only architecture is a sequence-model architecture that generates outputs using a decoder without a separate input encoder.
- How it is used
- Uses decoder-only transformers as the foundation for autoregressive generation, including text, code, and instruction-following tasks.
- How deep it goes
- Explains causal attention, token feedback, and next-token training, with listings for input encoding, masking, and a transformer block; abstracts the BPE implementation.
- Examples
- Illustrates input tokenization with a GPT-2 encoder and shows masked attention and transformer block listings.
- What you can do after
- Evaluate/understand how decoder-only models generate tokens and how their architecture differs from encoder–decoder and encoder-only variants.
Read how Transformers in Action explains decoder-only transformer →
Retrieval Augmented Generation, The Foundational Ideas
A decoder-only model generates text from a single autoregressive sequence without a separate encoder for input representation.
- How it is used
- Generates continuations from a single prompt sequence and supports modern text generation and long-context retrieval-augmented generation.
- How deep it goes
- Contrasts sequence processing, attention complexity, and middle-context failure risk, but does not descend into decoder internals.
- Examples
- Uses GPT, LLaMA, Mistral, and concatenated multi-document prompts as concrete decoder-only examples.
- What you can do after
- Evaluate decoder-only and encoder-decoder architectures for retrieval systems using context risk, complexity, and implementation tradeoffs.
Read how Retrieval Augmented Generation, The Foundational Ideas explains decoder-only transformer →
Building Specialized AI Systems
A decoder-only model generates sequences with decoder layers and causal attention over preceding tokens.
- How it is used
- Deploys decoder-only transformers for autoregressive text generation and selects them over encoder-only or encoder-decoder layouts when generation dominates.
- How deep it goes
- Explains causal masking, architectural alternatives, pretraining objectives, and quadratic attention costs, then points to mitigation techniques.
- Examples
- Names GPT-4, Llama, and Mistral as modern large language models using decoder-only architectures.
- What you can do after
- Evaluate whether a generation workload warrants decoder-only architecture and identify its context-length and adaptation trade-offs.
Read how Building Specialized AI Systems explains decoder-only transformer →
Build a Text-to-Image Generator (from Scratch)
A decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.
- How it is used
- Uses autoregressive next-token prediction for prompt-based text generation and image-token generation.
- How deep it goes
- Explains layer structure, autoregressive flow, and EOS termination, but does not detail decoder-layer internals.
- Examples
- Uses ChatGPT for text generation, translation decoding with EOS, and min-DALL-E for image-token generation.
- What you can do after
- Evaluate/understand how decoder-only models turn prompts into sequences through repeated next-token prediction.
Read how Build a Text-to-Image Generator (from Scratch) explains decoder-only transformer →
LLM Evaluation and Alignment, The Foundational Ideas
A decoder model is an autoregressive sequence model that predicts each token from preceding tokens and is typically used for text generation.
- How it is used
- Uses left-to-right prediction for autoregressive text generation.
- How deep it goes
- Contrasts left-to-right decoding with bidirectional encoding but does not implement decoder inference.
- Examples
- Uses GPT and Claude as generation-oriented examples and contrasts them with BERT-style encoders.
- What you can do after
- Evaluate whether bidirectional encoders or autoregressive decoders fit a text-comparison task.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains decoder-only transformer →
Sutskever's List
A transformer language-model architecture that generates sequences using only a decoder stack, without a separate encoder.
- How it is used
- Uses decoder-only transformers to study how architecture choice and scale affect language-model performance.
- How deep it goes
- Explains left-to-right conditioning through masked self-attention and compares long-context performance and computational efficiency.
- Examples
- Contrasts LSTMs' short-sequence performance with their plateau and uses GPT-4's Tom Cruise example to illustrate directional brittleness.
- What you can do after
- Evaluate architecture trade-offs in long-context performance and computational efficiency at scale.
Read how Sutskever's List explains decoder-only transformer →
How Large Language Models Work
A decoder-only transformer is a transformer architecture composed of decoder-style layers that generates outputs autoregressively from preceding context.
- How it is used
- Generates text by continuing preceding context one token at a time.
- How deep it goes
- Explains embeddings, position, attention, probabilities, and sampling, but does not derive transformer mathematics or training procedures.
- Examples
- Uses GPT, Gemini, ChatGPT, and the continuation of a partially written document as examples.
- What you can do after
- Evaluate generated text by recognizing next-token prediction, probability-based output, and sampling randomness.
Read how How Large Language Models Work explains decoder-only transformer →
Effective Conversational AI
A decoder-only architecture is a language-model architecture trained to predict successive tokens and generate text from preceding context.
- How it is used
- Supports conversational generation when responses must address user requests with specificity or personalization.
- How deep it goes
- Defines next-token prediction, training style, and task range, but does not descend into transformer internals.
- Examples
- Uses conversational responses, classification, question answering, summarization, and next-token text completion as examples.
- What you can do after
- Evaluate whether generative modeling fits conversational tasks requiring responses beyond predefined intent categories.
Read how Effective Conversational AI explains decoder-only transformer →
Deep Learning with PyTorch, Second Edition
A transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.
- How it is used
- Generates contextually relevant text and supports the dominant architecture used by modern generative language applications.
- How deep it goes
- Builds a GPT-style decoder in PyTorch with embeddings, repeated blocks, attention projections, normalization, and output projection.
- Examples
- Demonstrates character-level name generation using a compact PyTorch model and displays generated sample names.
- What you can do after
- Build it from scratch in PyTorch and generate short text samples with a GPT-style decoder.
Read how Deep Learning with PyTorch, Second Edition explains decoder-only transformer →
Also covered in
- Deep Learning with R, Third Edition
- Crack Any Codebase with AIIn early access (MEAP)
- Build an Advanced RAG Application (From Scratch)In early access (MEAP)
- Machines that ThinkIn early access (MEAP)
- LLMs in Production
What to know first
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
Related concepts
- GPT modelincludes · A GPT model is a generative, autoregressive Transformer language model in the family developed by OpenAI.
- Moirai-MoEused by · Moirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.
- TimesFMused by · TimesFM is a foundation model designed for deterministic time-series forecasting.
- transformer architecturekind of · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- encoder-only transformercontrasts with · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
- masked attentionuses · Masked attention is an attention mechanism that applies a mask to restrict which sequence positions can contribute to each position, often excluding future positions.
Questions about decoder-only transformer
What is decoder-only transformer?
A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.
Which Manning books explain decoder-only transformer?
Learn Generative AI with PyTorch, CUDA for LLMs, Fight Fraud with Machine Learning, Time Series Forecasting Using Foundation Models, Transformers in Action, Retrieval Augmented Generation, The Foundational Ideas, Building Specialized AI Systems, Build a Text-to-Image Generator (from Scratch) and 5 more.
What should I know before learning decoder-only transformer?
Manning books that teach decoder-only transformer usually introduce attention, language model, transformer architecture and BERT first.
What is the difference between decoder-only transformer and encoder-only transformer?
A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context. An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
What kinds of decoder-only transformer are there?
Kinds of decoder-only transformer include GPT model.
What is decoder-only transformer a kind of?
Decoder-only transformer is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.












