Decoder layer
A decoder layer is a Transformer block that transforms target-sequence representations using masked self-attention, feed-forward processing, and optionally cross-attention to encoder outputs.
Learn Generative AI with PyTorch
A Transformer building block composed of self-attention and feed-forward components, plus normalization and residual connections in this section.
- How it is used
- Forms the repeatable decoder unit for next-token prediction in translation, text generation, and music generation.
- How deep it goes
- Works through masked translation inputs and PyTorch forward passes, then shows how blocks are stacked into complete decoders.
- Examples
- Uses a five-token French target sequence, a 48-block GPT-2XL, and smaller text and music Transformers.
- What you can do after
- Build it from scratch in PyTorch as either an encoder–decoder layer or a decoder-only block, then stack it in a Transformer.
Read how Learn Generative AI with PyTorch explains decoder layer →
Inside Deep Learning: Math, Algorithms, Models
A transformer block that uses encoder outputs in a second attention stage for sequence generation.
- How it is used
- Supports sequence-to-sequence and autoregressive models that generate multiple outputs.
- How deep it goes
- Explains attention routing, residual structure, positional information, and future masking, but stops short of decoder implementation code.
- Examples
- Uses machine translation and sequence-to-sequence generation as application examples.
- What you can do after
- Understand how encoder outputs enter decoder attention and why autoregressive decoding requires a triangular mask.
Read how Inside Deep Learning: Math, Algorithms, Models explains decoder layer →
Related concepts
- attentionuses · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- cross-attentionuses · Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.
- transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- decodepart of · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
- decoder-only transformerpart of · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
- masked attentionuses · Masked attention is an attention mechanism in which a mask restricts which positions can influence each other, often preventing sequence positions from accessing future tokens.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.