A transformer processes a sequence by letting its tokens attend to one another, rather than passing information through recurrent neurons one token at a time. This makes computations across a sequence more parallelizable than in an RNN. For autoregressive text generation, however, the model still produces a token, feeds it back as input, and runs again to predict the next one. section 9.1 section 9.1.1
Three pieces work together to make that sequence processing useful:
- Byte pair encoding turns text into tokens using character-sequence statistics, keeping the vocabulary manageable. section 9.1.1
- Attention connects relevant tokens across the sequence. Multi-head attention is used within transformer layers. section 9.1.1 section 9.1.2
- Positional encoding preserves information about where tokens occur—information that removing recurrence would otherwise leave out. section 9.1.1
Think of transformer layers as matching LEGO bricks: each layer passes on a sequence encoding with the same tensor shape. Encoder layers can therefore be stacked without redesigning their interfaces; decoder layers likewise pass on consistently shaped representations. More layers provide progressively deeper representations, while attention within a layer can span the input sequence. The fixed-length sequence of embeddings used by attention also limits how much text a model can process at once. section 9.1.2 section 9.1.1
Figure 9.2 Consistent encoding shapes make transformers stackable.
The diagram shows why matching encoding shapes let one transformer layer connect directly to the next. section 9.1.2
Build and inspect a translation model
A practical translation pipeline loads aligned German–English sentences, inspects and splits the data, trains tokenizers, and extends PyTorch’s nn.Transformer with embeddings and positional encoding. Source padding masks exclude padding; a target mask controls which positions the decoder can use. Before training, a forward pass on random tensors checks the output shape. Training and evaluation then lead to sentence-level translations and plots of saved decoder attention weights. One test example translates a sentence about a mother and her son; the plotted weights offer a way to inspect the model’s token connections, not a guarantee that every translation is correct. section 9.2.3
The architecture supports tasks beyond translation, including text generation and language understanding. Choosing a pretrained model and fine-tuning it can make demanding applications feasible on modest hardware. Outputs still need evaluation: grammatical or plausible text is not, by itself, evidence of reliability, and automated content work has consequences for quality and employment. section 9.5 section 9.0
