Word order matters in translation. A transformer’s attention connects tokens without recurrence, so token embeddings need position information before they pass through its layers. A positional embedding is a vector representing a token’s place in the input sequence; adding it lets the model use both token identity and location. section 9.1.1 section 9.2.1
Position information can be learned or fixed. In either case, the position vector must have the same dimension as the token embedding so the two can be summed. A fixed sinusoidal encoding uses sine and cosine at different frequencies. Its design makes an encoding at an offset expressible as a linear function of the original encoding, helping the model learn relative positions. The sinusoidal values are calculated, not learned positional embeddings. section 9.2.1
Listing 9.1 Pytorch PositionalEncoding
>>> import math
>>> import torch
>>> from torch import nn
...
>>> class PositionalEncoding(nn.Module):
... def __init__(self, d_model=512, dropout=0.1, max_len=5000):
... super().__init__()
... self.dropout = nn.Dropout(p=dropout) #1
... self.d_model = d_model #2
... self.max_len = max_len #3
... pe = torch.zeros(max_len, d_model) #4
... position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
... div_term = torch.exp(torch.arange(0, d_model, 2).float() *
... (-math.log(10000.0) / d_model))
... pe[:, 0::2] = torch.sin(position * div_term) #5
... pe[:, 1::2] = torch.cos(position * div_term)
... pe = pe.unsqueeze(0).transpose(0, 1)
... self.register_buffer('pe', pe)
...
... def forward(self, x):
... x = x + self.pe[:x.size(0), :] #6
... return self.dropout(x)
Inspect the PyTorch PositionalEncoding module to see how it constructs sine and cosine values across positions and embedding dimensions. section 9.2.1
Add position information to a translation model
In a PyTorch translation transformer, create separate source and target token embeddings, then apply PositionalEncoding to each embedded sequence before passing it to torch.nn.Transformer. The source and target preparation methods also rearrange batch-first token IDs into sequence-first tensors. Check those shapes: a dimension error can disrupt the pipeline, sometimes without an obvious warning. Positional encoding supplies location; padding and subsequent-token masks serve separate purposes. section 9.2.3
BERT illustrates a learned-embedding arrangement: each input representation sums a WordPiece token embedding, a segment embedding identifying sentence A or B, and a positional embedding. Position is one part of the representation, not a replacement for token or segment information. section 9.3.1
Figure 9.11 Combining token semantics, context semantics, and token position information
Use the diagram to trace the three embeddings summed for each BERT input position. section 9.3.1
