Attention relates pairs of tokens, but it does not know whether a token came first or last. In a translation model, rearranging the source words could therefore leave its predictions unchanged. A positional embedding supplies the missing order information by adding a representation of each token’s position to its token embedding. section 15.3.4 section 15.3.5
Give each word a place
For a sentence’s first, second, and third tokens, look up position vectors for indices 0, 1, and 2. Add each position vector to the corresponding word vector. The resulting vectors carry both token identity and location before they enter the Transformer. section 15.3.5
A simple rule: learn position vectors with an embedding layer, then add them to token vectors of the same size. Using a raw position number as an extra input is less attractive: large integers and discrete input values can be troublesome for neural networks. Cyclic vectors built from cosine functions are another approach, but the learned lookup is the approach used here. section 15.3.5
Listing 15.17 A learned position embedding layer
from keras import ops class PositionalEmbedding(keras.Layer): def __init__(self, sequence_length, input_dim, output_dim): super().__init__() self.token_embeddings = layers.Embedding(input_dim, output_dim) self.position_embeddings = layers.Embedding(sequence_length, output_dim) def call(self, inputs): positions = ops.cumsum(ops.ones_like(inputs), axis=-1) - 1 #1 embedded_tokens = self.token_embeddings(inputs) embedded_positions = self.position_embeddings(positions) return embedded_tokens + embedded_positions
The Keras layer generates zero-based position indices, looks up token and position vectors separately, and adds them. section 15.3.5
For sequence-to-sequence translation, use position-aware embeddings on both the source entering the encoder and the target entering the decoder. With positional information added, the translation model reached 67% next-word accuracy. The same learned-position idea also appears in a decoder-only text generator; its embedding layer additionally supports a reverse projection using the token-embedding weights. section 15.3.5 section 16.2.1
Listing 15.18 Building a Transformer model with positional embeddings
hidden_dim = 256 intermediate_dim = 2056 num_heads = 8 source = keras.Input(shape=(None,), dtype="int32", name="english") x = PositionalEmbedding(sequence_length, vocab_size, hidden_dim)(source) encoder_output = TransformerEncoder(hidden_dim, intermediate_dim, num_heads)( source=x, source_mask=source != 0, ) target = keras.Input(shape=(None,), dtype="int32", name="spanish") x = PositionalEmbedding(sequence_length, vocab_size, hidden_dim)(target) x = TransformerDecoder(hidden_dim, intermediate_dim, num_heads)( target=x, source=encoder_output, source_mask=source != 0, ) x = layers.Dropout(0.5)(x) target_predictions = layers.Dense(vocab_size, activation="softmax")(x) transformer = keras.Model([source, target], target_predictions)
Shows where positional embeddings replace ordinary token embeddings in the translation encoder and decoder. section 15.3.5
