A positional embedding, also called a positional encoding, is a vector that tells a transformer where an element sits in a sequence. Self-attention processes elements without inherently recognizing their order. Adding a position vector to each token embedding gives the model both the element’s content and its position before attention begins. section 2.1.2
From words to ordered input
Tokenize “Wie geht es dir” into four tokens and represent each with a 256-value word embedding. Create a matching 4 × 256 positional-encoding matrix, then add the matrices element-wise. The resulting 4 × 256 tensor is the encoder input. section 2.1.1
Two implementations provide position vectors:
- Fixed encoding computes each vector with sine values at even indices and cosine values at odd indices. The values stay constant during training and add no learnable position parameters. section 2.2.3 section 3.1.1
- Learned encoding makes the position vectors trainable parameters. For image patches, this lets the vectors adapt to the task and dataset rather than follow a predetermined pattern. section 3.1.1
Listing 2.3 Designing the PositionalEncoding() class
class PositionalEncoding(nn.Module):
def __init__(self, d_model, dropout, max_len=5000): #1
super().__init__()
self.dropout = nn.Dropout(p=dropout)
pe = torch.zeros(max_len, d_model, device=DEVICE)
position = torch.arange(0., max_len,
device=DEVICE).unsqueeze(1)
div_term = torch.exp(torch.arange(
0., d_model, 2, device=DEVICE)
* -(math.log(10000.0) / d_model))
pe_pos = torch.mul(position, div_term)
pe[:, 0::2] = torch.sin(pe_pos) #2
pe[:, 1::2] = torch.cos(pe_pos) #3
pe = pe.unsqueeze(0)
self.register_buffer('pe', pe)
def forward(self, x):
x = x + self.pe[:, :x.size(1)].requires_grad_( #4
False) #4
out = self.dropout(x)
return out
Follow the PyTorch implementation: allocate one row per position, fill alternating dimensions with sine and cosine, register the result as a buffer, and add the needed rows to the input before dropout. section 2.2.3
The same addition works for a vision transformer. A CIFAR-10 image becomes 64 patch tokens plus a class token; learned position vectors match the resulting 65 × 48 embedding tensor. Fixed encoding is also used for image embeddings in a captioning transformer and to embed diffusion time steps in a denoising U-Net. section 3.3.2 section 7.3.1
Position information is not a substitute for geometry. In text-to-image generation, positional encodings can help represent spatial relationships but do not enforce consistent depth, perspective, or complex 3D constraints. section 1.5.2
