A transformer processes tokens in parallel, so it does not inherently know which token came first. A positional embedding supplies that missing order information. Adding it to each token embedding lets the model use position when interpreting a sequence, rather than treating the tokens as an unordered set. section 9.6.1
Position information can come from learned parameters or fixed sinusoidal functions. It can represent absolute or relative positions. In a decoder using learned embeddings, the model looks up one position vector per sequence position and adds those vectors to the token embeddings before the transformer blocks run. section 9.6.1
# In[]:
class Transformer(nn.Module):
def __init__(self, n_embd, vocab_size, block_size, num_blocks=6):
super().__init__()
self.char_embedding = nn.Embedding(vocab_size, n_embd)
self.positional_embedding = nn.Embedding(block_size, n_embd)
self.transformer_blocks = nn.Sequential(
*[TransformerBlock(n_embd) for _ in range(num_blocks)]
)
self.output_proj = nn.Linear(n_embd, vocab_size)
def forward(self, x):
_, seq_len = x.shape
pos_embd = self.positional_embedding(torch.arange(seq_len))
char_embd = self.char_embedding(x)
x = char_embd + pos_embd
x = self.transformer_blocks(x)
x = self.output_proj(x)
return x
n_embd = 64
model = Transformer(n_embd, vocab_size, block_size=max_name_length)
optimizer = optim.SGD(model.parameters(), lr=0.1)
train(model, optimizer, num_steps=10_001, loss_report_interval=1_000)
# Out[]:
Average loss at step 1000: 2.2924
Average loss at step 2000: 2.1520
Average loss at step 3000: 2.0999
Average loss at step 4000: 2.0626
Average loss at step 5000: 2.0346
Average loss at step 6000: 2.0107
Average loss at step 7000: 1.9852
Average loss at step 8000: 1.9629
Average loss at step 9000: 1.9477
Average loss at step 10000: 1.9299
The PyTorch decoder creates token and positional embedding layers. In forward, it looks up positions with torch.arange(seq_len), adds the result to the token embeddings, and passes the sum through transformer blocks. section 9.6.1
The same need appears with images. A Vision Transformer treats image patches as tokens, then adds learnable positional embeddings to encode patch locations before the encoder blocks process them. section 9.9
A related use of sinusoidal embeddings
A diffusion denoising model embeds its timestep and the two coordinates of each 2D point with sinusoidal embeddings. Here the technique supplies information about time and coordinates, rather than word order, before a network predicts noise. section 10.6
