A Transformer models a sequence without recurrent layers. Attention lets each token draw on other relevant tokens, so information can travel across a sentence without being passed along one step at a time. Unlike an RNN, its attention computations do not require a sequence-length loop during training, making them easier to parallelize. section 15.3 section 15.6
Consider translating English into Spanish. An encoder builds a representation of the English sentence; a decoder uses that representation and the Spanish words already available to predict the next word. The encoder uses bidirectional attention, while the decoder uses causal attention for next-word prediction. A comparable translation model can replace GRU layers with Transformer encoder and decoder blocks. section 15.3.4 section 15.6
Three implementation choices make the architecture work:
- Combine attention with a two-layer feedforward network: attention exchanges information across tokens, and the feedforward network updates each token’s representation. Encoder blocks also use residual connections and layer normalization. section 15.3.2
- Mask positions that should not contribute to attention scores, including masked source tokens in the encoder implementation. section 15.3.2
- Add position information to token embeddings. Attention alone treats tokens as a set: rearranging them does not tell it which word came first. section 15.3.4 section 15.3.5
The missing-position test
An English-to-Spanish Transformer without position embeddings reached about 58% next-word accuracy, worse than its GRU comparison. Adding learned position embeddings raised accuracy to 67%. In this experiment, each training epoch also took about a third of the RNN model’s time. The contrast makes word order an implementation requirement, not a decorative extra. section 15.3.4 section 15.3.5
Listing 15.17 A learned position embedding layer
layer_positional_embedding <- new_layer_class(
"PositionalEmbedding",
initialize = function(sequence_length, input_dim, output_dim) {
super$initialize()
self$token_embeddings <- layer_embedding(input_dim = input_dim,
output_dim = output_dim)
self$position_embeddings <- layer_embedding(input_dim = sequence_length,
output_dim = output_dim)
},
call = function(inputs) {
.[.., sequence_length] <- op_shape(inputs)
positions <-
op_arange(0, sequence_length - 1, dtype = "int32") |> #1
op_expand_dims(1)
embedded_tokens <- self$token_embeddings(inputs)
embedded_positions <- self$position_embeddings(positions)
embedded_tokens + embedded_positions
}
)
The R/Keras layer shows the central fix: look up a learned vector for each token position and add it to that token’s embedding. section 15.3.5
Transformers can also be pretrained for text classification or used for language modeling. They scale well with data and model size, but are data-hungry. They are not limited to text: an image encoder can turn image patches into vectors that join a text-token sequence. section 15.4 section 16.4.2
