liveBook · Manning

Decoder layer

Also called decoder block

A decoder layer is a neural-network component that transforms representations to produce an output sequence, often using self-attention and, in encoder–decoder models, information from an encoder.

How 2 Manning books explain it

Learn Generative AI with PyTorch

A Transformer building block composed of self-attention and feed-forward components, plus normalization and residual connections in this section.

How it is used
Uses decoder layers as the stacked core of translation, text-generation, and music-generation Transformers.
How deep it goes
Works through masked and cross-attention, residual connections, normalization, feed-forward dimensions, and decoder stacking in code.
Examples
Works through a five-token French translation and decoder blocks for GPT-2XL, a smaller text model, and a music Transformer.
What you can do after
Builds decoder layers from scratch and stacks them into encoder-decoder or decoder-only Transformer models.

Read how Learn Generative AI with PyTorch explains decoder layer →

Inside Deep Learning: Math, Algorithms, Models

A transformer block that uses encoder outputs in a second attention stage for sequence generation.

How it is used
Uses encoder context to support sequence generation, including tasks with multiple outputs.
How deep it goes
Describes the attention stages and future-leakage risk, without detailing a decoder implementation.
Examples
Names machine translation and sequence-to-sequence or autoregressive models as applications.
What you can do after
Understand how encoder outputs enter a decoder and why autoregressive generation needs a future mask.

Read how Inside Deep Learning: Math, Algorithms, Models explains decoder layer →

Also covered in

What to know first

Related concepts

Questions about decoder layer

What is decoder layer?

A decoder layer is a neural-network component that transforms representations to produce an output sequence, often using self-attention and, in encoder–decoder models, information from an encoder.

Which Manning books explain decoder layer?

Learn Generative AI with PyTorch and Inside Deep Learning: Math, Algorithms, Models.

What should I know before learning decoder layer?

Manning books that teach decoder layer usually introduce positional embedding, bidirectional rnn, encoder and feedforward network first.

What is decoder layer part of?

Decoder layer is part of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.

What are the parts of decoder layer?

Decoder layer includes feedforward network.

What is decoder layer also called?

Decoder layer is also called decoder block.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.