A Transformer models a sequence without passing a recurrent state from token to token. Instead, attention lets a token draw information selectively from other tokens. This helps with long-range dependencies that can trouble RNNs, and removes the sequence-length loop during training, making computation easier to parallelize. section 15.2.2 section 15.6
English-to-Spanish translation
Consider predicting the next Spanish token from an English sentence and the Spanish tokens already generated. An encoder represents the English source; a decoder uses that representation and past target tokens to make its prediction. In a Transformer, attention replaces the GRU layers that pass information across each sequence. section 15.3.4 section 15.6
The working parts have distinct jobs:
- Self-attention passes information between tokens. An encoder uses bidirectional attention to represent a source sequence; a decoder uses causal attention when predicting the next token. section 15.3.2 section 15.6
- A two-layer feedforward network updates each token’s representation after attention passes information across the sequence. Residual connections and layer normalization are also used in the encoder block. section 15.3.2
- Attention masks keep padding tokens out of the calculation. Positional embeddings add word-position information: attention alone treats tokens as a set and cannot tell whether a word came first or last. section 15.3.2 section 15.3.4 section 15.3.5
Listing 15.16 Building a Transformer model
hidden_dim = 256 intermediate_dim = 2048 num_heads = 8 source = keras.Input(shape=(None,), dtype="int32", name="english") x = layers.Embedding(vocab_size, hidden_dim)(source) encoder_output = TransformerEncoder(hidden_dim, intermediate_dim, num_heads)( source=x, source_mask=source != 0, ) target = keras.Input(shape=(None,), dtype="int32", name="spanish") x = layers.Embedding(vocab_size, hidden_dim)(target) x = TransformerDecoder(hidden_dim, intermediate_dim, num_heads)( target=x, source=encoder_output, source_mask=source != 0, ) x = layers.Dropout(0.5)(x) target_predictions = layers.Dense(vocab_size, activation="softmax")(x) transformer = keras.Model([source, target], target_predictions)
The translation model wires source and target embeddings to a Transformer encoder and decoder, then predicts target-token probabilities with a Dense layer. Notice the source mask passed to both blocks. section 15.3.4
Word position is an easy detail to miss. In the translation experiment, a Transformer without positional information reached about 58% next-word accuracy. Adding learned positional embeddings raised it to 67%; the resulting model also trained faster per epoch than the GRU comparison. The practical rule: if order matters, supply position information rather than expecting attention to discover it from token embeddings alone. section 15.3.4 section 15.3.5
Transformers can scale with more data, layers, and parameters, but they are data-hungry. For a small text-classification problem or limited compute budget, a fast set-based model may be more cost-effective. Large language models combine the architecture with next-token prediction and large amounts of unlabeled text; the architecture alone is not the whole recipe. section 15.4 section 14.5.5 section 16.6
