How deeply does Deep Learning with PyTorch, Second Edition cover decoder-only transformer?

Builds a GPT-style decoder in PyTorch with embeddings, repeated blocks, attention projections, normalization, and output projection.

Decoder-only transformer

As explained in Deep Learning with PyTorch, Second Edition

A transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.

A decoder-only transformer generates a sequence of tokens from an input sequence using only the transformer’s decoding stack. Unlike the original encoder–decoder design, it does not use a separate encoder to provide representations. GPT-style models, including GPT, Claude, and Llama, use this architecture for coherent text generation. section 9.12 section 9.6

A decoder block combines several mechanisms:

Multiheaded attention uses parallel query, key, and value projections so different heads can capture different relationships in the sequence. section 9.6.1
Residual connections let gradients flow around layers, while layer normalization stabilizes the block. section 9.6.1
Feed-forward layers extend the attention block before repeated decoder blocks transform the sequence representation. section 9.6.1

A small GPT-style model

A PyTorch implementation embeds characters and positions, passes their sum through repeated TransformerBlock instances, and projects the result to vocabulary logits. Training the model on names can produce samples such as “journe,” “huxson,” and “bailanie.”

How Deep Learning with PyTorch, Second Edition covers it

Cover of Deep Learning with PyTorch, Second Edition

liveBook manning.com

Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Learn Generative AI with PyTorchA Transformer design that uses only decoder stacks for autoregressive generation.
liveBook manning.com

Time Series Forecasting Using Foundation ModelsA decoder-only transformer is a transformer architecture that generates sequence outputs autoregressively from previously generated tokens without an encoder component.
liveBook manning.com

Transformers in ActionA decoder-only architecture is a sequence-model architecture that generates outputs using a decoder without a separate input encoder.
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)A decoder-only transformer consists of decoder layers that autoregressively generate output sequences from preceding tokens.
liveBook manning.com

Sutskever's ListA transformer language-model architecture that generates sequences using only a decoder stack, without a separate encoder.
liveBook manning.com

How Large Language Models WorkA decoder-only transformer is a transformer architecture composed of decoder-style layers that generates outputs autoregressively from preceding context.
liveBook manning.com

Effective Conversational AIA decoder-only architecture is a language-model architecture trained to predict successive tokens and generate text from preceding context.

Compare how 8 books explain decoder-only transformer →

Related concepts

GPT modelincludes · A GPT model is a generative pretrained Transformer language model that produces text autoregressively by predicting successive tokens from preceding context.
TimesFMused by · TimesFM is a foundation model designed for deterministic time-series forecasting.
Moirai-MoEused by · Moirai-MoE is a Moirai-based universal time-series forecasting model that uses fixed-length, nonoverlapping patches across frequencies.
transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
encoder-only transformercontrasts with · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
masked attentionuses · Masked attention is an attention mechanism in which a mask restricts which positions can influence each other, often preventing sequence positions from accessing future tokens.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.