Transformers process tokens in parallel, so token embeddings alone do not tell the model their order. A positional embedding adds position-specific information to each token embedding, helping the model use order when interpreting a sequence. section 9.6.1
Positional embeddings can be learned parameters or fixed sinusoidal functions. They can represent absolute or relative position information. In a learned absolute-position approach, each position has an embedding that is added to the token embedding at that position. section 9.6.1
Adding positions in a decoder
A character-level transformer uses one embedding table for characters and another for positions. It looks up positions from zero to the sequence length, adds those vectors to the character embeddings, and passes the result through transformer blocks. The same character at different positions is paired with a position-specific vector. section 9.6.1
# In[]:
class Transformer(nn.Module):
def __init__(self, n_embd, vocab_size, block_size, num_blocks=6):
super().__init__()
self.char_embedding = nn.Embedding(vocab_size, n_embd)
self.positional_embedding = nn.Embedding(block_size, n_embd)
self.transformer_blocks = nn.Sequential(
*[TransformerBlock(n_embd) for _ in range(num_blocks)]
)
self.output_proj = nn.Linear(n_embd, vocab_size)
def forward(self, x):
_, seq_len = x.shape
pos_embd = self.positional_embedding(torch.arange(seq_len))
char_embd = self.char_embedding(x)
x = char_embd + pos_embd
x = self.transformer_blocks(x)
x = self.output_proj(x)
return x
n_embd = 64
model = Transformer(n_embd, vocab_size, block_size=max_name_length)
optimizer = optim.SGD(model.parameters(), lr=0.1)
train(model, optimizer, num_steps=10_001, loss_report_interval=1_000)
# Out[]:
Average loss at step 1000: 2.2924
Average loss at step 2000: 2.1520
Average loss at step 3000: 2.0999
Average loss at step 4000: 2.0626
Average loss at step 5000: 2.0346
Average loss at step 6000: 2.0107
Average loss at step 7000: 1.9852
Average loss at step 8000: 1.9629
Average loss at step 9000: 1.9477
Average loss at step 10000: 1.9299
The decoder listing shows the position lookup and addition using PyTorch's nn.Embedding. section 9.6.1
Positions beyond text
A Vision Transformer adds learned positional embeddings to image-patch tokens to encode their locations before the encoder blocks. Sinusoidal embeddings can also encode a timestep and 2D coordinates for a diffusion model's noise-prediction network. section 9.9 section 10.6



















