A transformer has no recurrence to indicate where a token occurs in a sequence. Positional encoding supplies that missing order information: without it, the model could not distinguish token order reliably. The encoding is added to each input token embedding before attention, so the resulting representation carries both token identity and position. section 2.2.2 section 2.2.3
One implementation uses sine and cosine waves to encode positions across embedding dimensions. Their frequencies vary by dimension, giving different positions distinct patterns. A PyTorch implementation creates a position-by-hidden-dimension tensor, fills alternating dimensions with sine and cosine values, and registers it as a buffer. During the forward pass, it slices the tensor to the input sequence length, adds it to the input embeddings, and applies dropout. section 2.2.2
Listing 2.2 Example implementation for positional encoding
class PositionalEncoding(nn.Module):
"""Positional encoding class."""
def init (self, num_hiddens, max_len=1000):
super(). init ()
self.dropout = nn.Dropout(0.1)
# Create a positional embedding matrix
position = torch.arange(max_len, dtype=torch.float32).reshape(-1, 1)
div_term = torch.exp(
torch.arange(0, num_hiddens, 2, dtype=torch.float32) *
-(math.log(10000.0) / num_hiddens))
pe = torch.zeros((1, max_len, num_hiddens))
pe[0, :, 0::2] = torch.sin(position * div_term)
pe[0, :, 1::2] = torch.cos(position * div_term)
self.register_buffer('pe', pe)
def forward(self, X):
X = X + self.pe[:, :X.shape[1], :].to(X.device)
return self.dropout(X)
The PositionalEncoding listing shows the alternating sine/cosine tensor, its registered buffer, and the sequence-length slice used when adding it to an input. section 2.2.2
The same need arises with images. A vision transformer flattens image patches, projects them into embeddings, and adds positional embeddings so the resulting one-dimensional sequence retains information about the patches’ original spatial arrangement. A classification token also receives a positional embedding. The visual description does not specify that these positional embeddings use the sinusoidal construction above. section 6.3.1
Figure 6.5 Architectural overview of patching, flattening and projecting, and adding positional and classification embeddings in the ViT. Image taken from [1].
The vision-transformer architecture diagram places positional embeddings alongside patch projection and the classification token. section 6.3.1
