liveBook · Manning

Decoder-only transformer

Also called decoder-only model, decoder-only architecture

A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.

How 13 Manning books explain it

Learn Generative AI with PyTorch

A Transformer design that uses only decoder stacks for autoregressive generation.

How it is used
Uses decoder-only transformers for autoregressive text and music generation from preceding tokens.
How deep it goes
Explains generation and decoder-layer architecture, then specifies embeddings, attention heads, vocabulary, and sequence length for a music model.
Examples
Uses GPT-2XL prompt completion and a six-layer music transformer operating on MIDI event tokens.
What you can do after
Builds decoder-only models from scratch and loads pretrained GPT-2 weights to generate text.

Read how Learn Generative AI with PyTorch explains decoder-only transformer →

CUDA for LLMs

A decoder-only transformer is a transformer architecture that generates sequences using causal attention over preceding tokens.

How it is used
Uses decoder-only transformers as the baseline workload for character-level modeling and CUDA inference.
How deep it goes
Traces the model from embeddings through masked attention, residual blocks, logits, and loss, then introduces inference phases and caching.
Examples
Uses a 65-character vocabulary and an eight-block model trained on The Wonderful Wizard of Oz.
What you can do after
Build a character-level transformer forward pass in PyTorch with masked attention and vocabulary logits.

Read how CUDA for LLMs explains decoder-only transformer →

Fight Fraud with Machine Learning

A decoder-only architecture is a generative sequence-modeling design containing decoder blocks without a separate encoder component.

How it is used
Uses decoder blocks to generate prompt-conditioned text one token at a time.
How deep it goes
Explains causal attention, iterative next-token prediction, removed encoder-decoder attention, and byte-pair encoding without decoder-block implementation details.
Examples
Illustrates next-word prediction, greedy search, English-to-French translation, and GPT autoregressive generation diagrams.
What you can do after
Evaluate/understand why GPT generates sequentially and differs structurally from an encoder-decoder translation model.

Read how Fight Fraud with Machine Learning explains decoder-only transformer →

Time Series Forecasting Using Foundation Models

A decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.

How it is used
Generates time-series forecasts autoregressively within probabilistic and deterministic foundation models.
How deep it goes
Explains patch tokens, sequential generation, self-attention, output patches, and expert routing, while stopping short of full implementation details.
Examples
Uses Lag-Llama, Moirai-MoE, and TimesFM to contrast probabilistic outputs, expert routing, and long-horizon output patches.
What you can do after
Evaluate decoder-only forecasting designs by comparing token granularity, autoregressive steps, output type, and architectural specialization.

Read how Time Series Forecasting Using Foundation Models explains decoder-only transformer →

Transformers in Action

A decoder-only architecture is a sequence-model architecture that generates outputs using a decoder without a separate input encoder.

How it is used
Uses decoder-only transformers as the foundation for autoregressive generation, including text, code, and instruction-following tasks.
How deep it goes
Explains causal attention, token feedback, and next-token training, with listings for input encoding, masking, and a transformer block; abstracts the BPE implementation.
Examples
Illustrates input tokenization with a GPT-2 encoder and shows masked attention and transformer block listings.
What you can do after
Evaluate/understand how decoder-only models generate tokens and how their architecture differs from encoder–decoder and encoder-only variants.

Read how Transformers in Action explains decoder-only transformer →

Retrieval Augmented Generation, The Foundational Ideas

A decoder-only model generates text from a single autoregressive sequence without a separate encoder for input representation.

How it is used
Generates continuations from a single prompt sequence and supports modern text generation and long-context retrieval-augmented generation.
How deep it goes
Contrasts sequence processing, attention complexity, and middle-context failure risk, but does not descend into decoder internals.
Examples
Uses GPT, LLaMA, Mistral, and concatenated multi-document prompts as concrete decoder-only examples.
What you can do after
Evaluate decoder-only and encoder-decoder architectures for retrieval systems using context risk, complexity, and implementation tradeoffs.

Read how Retrieval Augmented Generation, The Foundational Ideas explains decoder-only transformer →

Building Specialized AI Systems

A decoder-only model generates sequences with decoder layers and causal attention over preceding tokens.

How it is used
Deploys decoder-only transformers for autoregressive text generation and selects them over encoder-only or encoder-decoder layouts when generation dominates.
How deep it goes
Explains causal masking, architectural alternatives, pretraining objectives, and quadratic attention costs, then points to mitigation techniques.
Examples
Names GPT-4, Llama, and Mistral as modern large language models using decoder-only architectures.
What you can do after
Evaluate whether a generation workload warrants decoder-only architecture and identify its context-length and adaptation trade-offs.

Read how Building Specialized AI Systems explains decoder-only transformer →

Build a Text-to-Image Generator (from Scratch)

A decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.

How it is used
Uses autoregressive next-token prediction for prompt-based text generation and image-token generation.
How deep it goes
Explains layer structure, autoregressive flow, and EOS termination, but does not detail decoder-layer internals.
Examples
Uses ChatGPT for text generation, translation decoding with EOS, and min-DALL-E for image-token generation.
What you can do after
Evaluate/understand how decoder-only models turn prompts into sequences through repeated next-token prediction.

Read how Build a Text-to-Image Generator (from Scratch) explains decoder-only transformer →

LLM Evaluation and Alignment, The Foundational Ideas

A decoder model is an autoregressive sequence model that predicts each token from preceding tokens and is typically used for text generation.

How it is used
Uses left-to-right prediction for autoregressive text generation.
How deep it goes
Contrasts left-to-right decoding with bidirectional encoding but does not implement decoder inference.
Examples
Uses GPT and Claude as generation-oriented examples and contrasts them with BERT-style encoders.
What you can do after
Evaluate whether bidirectional encoders or autoregressive decoders fit a text-comparison task.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains decoder-only transformer →

Sutskever's List

A transformer language-model architecture that generates sequences using only a decoder stack, without a separate encoder.

How it is used
Uses decoder-only transformers to study how architecture choice and scale affect language-model performance.
How deep it goes
Explains left-to-right conditioning through masked self-attention and compares long-context performance and computational efficiency.
Examples
Contrasts LSTMs' short-sequence performance with their plateau and uses GPT-4's Tom Cruise example to illustrate directional brittleness.
What you can do after
Evaluate architecture trade-offs in long-context performance and computational efficiency at scale.

Read how Sutskever's List explains decoder-only transformer →

How Large Language Models Work

A decoder-only transformer is a transformer architecture composed of decoder-style layers that generates outputs autoregressively from preceding context.

How it is used
Generates text by continuing preceding context one token at a time.
How deep it goes
Explains embeddings, position, attention, probabilities, and sampling, but does not derive transformer mathematics or training procedures.
Examples
Uses GPT, Gemini, ChatGPT, and the continuation of a partially written document as examples.
What you can do after
Evaluate generated text by recognizing next-token prediction, probability-based output, and sampling randomness.

Read how How Large Language Models Work explains decoder-only transformer →

Effective Conversational AI

A decoder-only architecture is a language-model architecture trained to predict successive tokens and generate text from preceding context.

How it is used
Supports conversational generation when responses must address user requests with specificity or personalization.
How deep it goes
Defines next-token prediction, training style, and task range, but does not descend into transformer internals.
Examples
Uses conversational responses, classification, question answering, summarization, and next-token text completion as examples.
What you can do after
Evaluate whether generative modeling fits conversational tasks requiring responses beyond predefined intent categories.

Read how Effective Conversational AI explains decoder-only transformer →

Deep Learning with PyTorch, Second Edition

A transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.

How it is used
Generates contextually relevant text and supports the dominant architecture used by modern generative language applications.
How deep it goes
Builds a GPT-style decoder in PyTorch with embeddings, repeated blocks, attention projections, normalization, and output projection.
Examples
Demonstrates character-level name generation using a compact PyTorch model and displays generated sample names.
What you can do after
Build it from scratch in PyTorch and generate short text samples with a GPT-style decoder.

Read how Deep Learning with PyTorch, Second Edition explains decoder-only transformer →

Also covered in

What to know first

Related concepts

Questions about decoder-only transformer

What is decoder-only transformer?

A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.

Which Manning books explain decoder-only transformer?

Learn Generative AI with PyTorch, CUDA for LLMs, Fight Fraud with Machine Learning, Time Series Forecasting Using Foundation Models, Transformers in Action, Retrieval Augmented Generation, The Foundational Ideas, Building Specialized AI Systems, Build a Text-to-Image Generator (from Scratch) and 5 more.

What should I know before learning decoder-only transformer?

Manning books that teach decoder-only transformer usually introduce attention, language model, transformer architecture and BERT first.

What is the difference between decoder-only transformer and encoder-only transformer?

A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context. An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.

What kinds of decoder-only transformer are there?

Kinds of decoder-only transformer include GPT model.

What is decoder-only transformer a kind of?

Decoder-only transformer is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.