liveBook · Manning

Positional embedding

As explained in Deep Learning with PyTorch, Second Edition

An embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.

Transformers process tokens in parallel, so token embeddings alone do not tell the model their order. A positional embedding adds position-specific information to each token embedding, helping the model use order when interpreting a sequence. section 9.6.1

Positional embeddings can be learned parameters or fixed sinusoidal functions. They can represent absolute or relative position information. In a learned absolute-position approach, each position has an embedding that is added to the token embedding at that position. section 9.6.1

Adding positions in a decoder

A character-level transformer uses one embedding table for characters and another for positions. It looks up positions from zero to the sequence length, adds those vectors to the character embeddings, and passes the result through transformer blocks. The same character at different positions is paired with a position-specific vector. section 9.6.1

# In[]:
class Transformer(nn.Module):
    def __init__(self, n_embd, vocab_size, block_size, num_blocks=6):
        super().__init__()
        self.char_embedding = nn.Embedding(vocab_size, n_embd)
        self.positional_embedding = nn.Embedding(block_size, n_embd)

        self.transformer_blocks = nn.Sequential(
            *[TransformerBlock(n_embd) for _ in range(num_blocks)]
        )

        self.output_proj = nn.Linear(n_embd, vocab_size)

    def forward(self, x):
        _, seq_len = x.shape

        pos_embd = self.positional_embedding(torch.arange(seq_len))
        char_embd = self.char_embedding(x)
        x = char_embd + pos_embd
        x = self.transformer_blocks(x)
        x = self.output_proj(x)

        return x

n_embd = 64
model = Transformer(n_embd, vocab_size, block_size=max_name_length)
optimizer = optim.SGD(model.parameters(), lr=0.1)
train(model, optimizer, num_steps=10_001, loss_report_interval=1_000)

# Out[]:
Average loss at step 1000: 2.2924
Average loss at step 2000: 2.1520
Average loss at step 3000: 2.0999
Average loss at step 4000: 2.0626
Average loss at step 5000: 2.0346
Average loss at step 6000: 2.0107
Average loss at step 7000: 1.9852
Average loss at step 8000: 1.9629
Average loss at step 9000: 1.9477
Average loss at step 10000: 1.9299

The decoder listing shows the position lookup and addition using PyTorch's nn.Embedding. section 9.6.1

Positions beyond text

A Vision Transformer adds learned positional embeddings to image-patch tokens to encode their locations before the encoder blocks. Sinusoidal embeddings can also encode a timestep and 2D coordinates for a diffusion model's noise-prediction network. section 9.9 section 10.6

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes
Explains learned position lookup and addition in a decoder, while omitting the sinusoidal embedding implementation.
Examples
Uses a character-level decoder, image-patch classification, and a 2D noise-prediction network as examples.
What you can do after
Apply it via PyTorch's nn.Embedding to add learned positions to transformer inputs.
How it connects
Connects token embeddings and transformer decoders with Vision Transformers, image patches, and diffusion models.

Also explained in

Compare how 20 books explain positional embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.