A transformer models relationships within a sequence using attention. For text generation, that means a prediction can respond to earlier tokens rather than relying only on a fixed rule. Transformers are also used for content understanding; their original encoder–decoder design was used for machine translation. section 9.12 section 9.6
The two main configurations serve different jobs:
- An encoder turns an input sequence into representations of relationships among its elements. A decoder uses those representations to produce an output sequence. In encoder–decoder attention, the encoder output supplies the decoder’s keys and values. section 9.7.2
- A decoder-only transformer generates text from its input context. Its blocks combine multiheaded attention, feed-forward layers, residual connections, and layer normalization. Positional embeddings supply token-order information that parallel processing does not provide on its own. section 9.6 section 9.6.1
Attention makes context useful: queries seek relevant positions, keys help determine relevance, and values contribute information to the result. In causal self-attention, a mask prevents a position from using later tokens when predicting what comes next. section 9.12 section 9.5.2
Generating names
A PyTorch decoder embeds characters, adds positional embeddings, passes them through transformer blocks, and projects the result to vocabulary scores. Sampling from the trained model produces new name-like character sequences, such as “journe” and “chanie.” section 9.6.1
# In[]:
class Transformer(nn.Module):
def __init__(self, n_embd, vocab_size, block_size, num_blocks=6):
super().__init__()
self.char_embedding = nn.Embedding(vocab_size, n_embd)
self.positional_embedding = nn.Embedding(block_size, n_embd)
self.transformer_blocks = nn.Sequential(
*[TransformerBlock(n_embd) for _ in range(num_blocks)]
)
self.output_proj = nn.Linear(n_embd, vocab_size)
def forward(self, x):
_, seq_len = x.shape
pos_embd = self.positional_embedding(torch.arange(seq_len))
char_embd = self.char_embedding(x)
x = char_embd + pos_embd
x = self.transformer_blocks(x)
x = self.output_proj(x)
return x
n_embd = 64
model = Transformer(n_embd, vocab_size, block_size=max_name_length)
optimizer = optim.SGD(model.parameters(), lr=0.1)
train(model, optimizer, num_steps=10_001, loss_report_interval=1_000)
# Out[]:
Average loss at step 1000: 2.2924
Average loss at step 2000: 2.1520
Average loss at step 3000: 2.0999
Average loss at step 4000: 2.0626
Average loss at step 5000: 2.0346
Average loss at step 6000: 2.0107
Average loss at step 7000: 1.9852
Average loss at step 8000: 1.9629
Average loss at step 9000: 1.9477
Average loss at step 10000: 1.9299
See how character and position embeddings feed stacked TransformerBlock modules before the output projection. section 9.6.1
Figure 9.13 Diagram of the Transformer architecture from the original paper “Attention Is All You Need.” The encoder is on the left, and the decoder is on the right. The encoder output is fed into the decoder as the key and value for the multi-head attention mechanism.
Compare the encoder and decoder, including the path that supplies encoder outputs as keys and values to decoder attention. section 9.7.2
