How deeply does Deep Learning with R, Third Edition cover transformer decoder?

Explains encoder context, attention, positional encoding, embeddings, feedforward dimensions, masking, and softmax outputs through a working architecture.

Transformer decoder

As explained in Deep Learning with R, Third Edition

A Transformer decoder is an architectural component that generates sequence representations or outputs using attention and feedforward processing.

A Transformer decoder generates target-sequence predictions by combining a target sequence with the context-aware output of a Transformer encoder. Attention replaces recurrent layers as the mechanism for passing information across the sequence, while positional encoding preserves word order. section 20.1.6 section 15.3.4

English-to-Spanish translation

Build an encoder with hidden size 256, feedforward dimension 2,048, and eight attention heads. Embed the English source sequence, pass it through the encoder, embed the Spanish target sequence, and pass both target embeddings and encoder output to the decoder. Apply dropout and a vocabulary-sized softmax to produce token predictions, then train with sparse categorical cross-entropy and accuracy.

A decoder is useful when:

The task maps one sequence to another, such as machine translation or question answering. section 20.1.6
The model must use encoder context while predicting the next target word. section 20.1.6
Attention should connect sequence elements without relying on recurrent computation. section 15.3.4

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Related concepts

transformer encoderused by · A Transformer encoder is a neural-network component that processes an input sequence using stacked self-attention and feedforward layers to produce contextual representations.
transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.