A Transformer is a neural network for sequences built around attention. Unlike a recurrent neural network, which processes elements one after another, it can process input positions in parallel. Self-attention assigns weights to relationships among positions, helping the model use information from far apart in a sequence. Because parallel processing alone does not tell the model where tokens occur, positional encodings are added to their embeddings. section 1.3 section 10.5
Choose the arrangement that matches the task:
- Encoder–decoder: an encoder turns an input sequence into vectors; a decoder uses those vectors and earlier output tokens to produce the next token. Translation is the running example. section 1.3.2 section 9.1.2
- Encoder-only: stacked encoder layers produce representations useful for tasks such as text classification; BERT is an example. section 9.1.3
- Decoder-only: stacked decoder layers repeatedly predict a next token from a prompt; GPT-style models use this arrangement for text generation. section 9.1.3
Figure 9.4 The Transformer architecture. The encoder in the Transformer (left side of the diagram), which consists of N identical encoder layers, learns the meaning of the input sequence and converts it into vectors that represent its meaning. It then passes these vectors to the decoder (right side of the diagram), which consists of N identical decoder layers. The decoder constructs the output (e.g., the French translation of an English phrase) by predicting one token at a time, based on previous tokens in the sequence and vector representations from the encoder. The generator on the top right is the head attached to the output from the decoder so that the output is the probability distribution over all tokens in the target language (e.g., the French vocabulary).
Trace the translation model from its stack of encoder layers through the decoder layers to the generator, which outputs probabilities over target-language tokens. section 9.1.2
To build the translation model, start with token indexes. Word embeddings plus positional encodings form the inputs to attention. In each encoder layer, multihead self-attention is followed by a feed-forward network; each sublayer has a residual connection and layer normalization. Each decoder layer adds a masked self-attention sublayer and a cross-attention sublayer that uses the encoder output, then a feed-forward network. The implementation assembles an encoder, decoder, source embedding, target embedding, and generator. section 9.1.1 section 9.1.2 section 9.3.2
From attention heads to a translation
For “How are you?”, adding beginning- and end-of-sequence tokens gives an embedding tensor of shape (1, 6, 256). Multihead attention splits its 256-dimensional query, key, and value vectors into eight heads of 32 dimensions each, calculates attention in each head, and joins the results. Once trained, the encoder–decoder model encodes an English phrase once, then decodes French tokens one at a time until the end token; “I don’t speak French” and “I do not speak French” produce the same reported translation. section 9.2.1 section 10.4 section 10.5
The architecture can also be scaled to the available compute: a roughly five-million-parameter GPT-style model uses three decoder blocks and 256-dimensional embeddings, rather than GPT-2XL’s 48 blocks and 1,600-dimensional embeddings. For music, notes must first become indexed event tokens such as note-on, note-off, time-shift, and velocity before they can enter a music Transformer. section 12.0 section 14.1.1
