Chapter 15: Language models and the Transformer

Introduction

This introduction previews text generation, English-to-Spanish translation, and the Transformer as the chapter’s topics.

15.1 The language model

The section explains how text generation becomes a tractable prediction problem by representing text as token sequences and predicting one next token at a time.

• Text as output token sequences: Explain how generated text can be represented as an integer sequence and converted back to readable text.
  Why it matters: This connects a model’s numeric outputs to the text a reader actually sees.
• Whole sequence classification explosion: Justify why treating every possible output sequence as a separate classifier output is impractical.
  Why it matters: It explains why text generation cannot simply reuse classification with one class per complete answer.
• Next token language modeling: Explain what a language model predicts and how repeated predictions can produce a longer text sequence.
  Why it matters: This is the basic prediction task that makes language-model text generation feasible.

Key terms: integer sequence, tokens, detokenize, vocabulary, output integer sequences, classifier output, language model, p(token|past tokens), probability distribution, next token

15.1.1 Training a Shakespeare language model

The section prepares Shakespeare text as character-level next-character training data, builds a small recurrent language model, and reports its training result.

• Next-character training pairs: Describe how the Shakespeare text becomes fixed-length training examples whose labels are the input characters shifted by one position.
  Why it matters: The one-character offset defines the prediction task the model is trained to perform.
• Character-level text preparation: Explain how TextVectorization is configured to turn the chunks into integer sequences, and describe the remaining dataset preparation.
  Why it matters: The text must be converted into consistently tokenized integer sequences before the model can train on it.
• Recurrent model design: Justify the use of an RNN for next-character prediction and describe the small model's embedding and recurrent layer.
  Why it matters: The architecture is chosen to use preceding characters while keeping the demonstration manageable.
• Per-character classification objective: Explain what the model predicts at each sequence position, how it is trained, and what result the section reports.
  Why it matters: It connects language modeling here to the familiar task of classification, repeated at every character position.

Key terms: equal-length chunks, feature and label sequences, next character, TextVectorization, character-level vocabulary, split, standardization, RNN, recurrent state, Embedding

15.1.2 Generating Shakespeare

This section turns a trained next-character Shakespeare model into an autoregressive generator by adapting its GRU state handling, priming it with a prompt, and feeding predictions back one character at a time.

• Single-character inference model: Explain why generation uses a modified version of the trained model that accepts one character and an explicit GRU state, and why the trained weights can be reused.
  Why it matters: This explains how the trained predictor can be used repeatedly without losing its context.
• Prompt-based state priming: Describe how feeding a prompt one token at a time prepares the GRU state and supplies the prediction used to start generation.
  Why it matters: Priming makes the generated continuation depend on the chosen opening text.
• Autoregressive character generation loop: Trace how repeated next-character predictions become an output sequence, including feedback of selected characters, persistence of state, and conversion of token IDs to text.
  Why it matters: The feedback loop turns a predictor trained for one next character into a generator of open-ended text.
• Forward-only training constraint: Explain why next-character training must prevent future characters from influencing a prediction and how generation differs from training.
  Why it matters: It clarifies why impressive training accuracy need not mean a model can generate usable text.

Key terms: GRU cell’s state, RNN state, autoregressive model, prompt, prime, internal state of the GRU, most likely next character, detokenize, Bidirectional(GRU(...)), training accuracy

15.2 Sequence-to-sequence learning

This section introduces sequence-to-sequence modeling and explains how an encoder and decoder work during training and inference.

• Sequence-to-sequence training setup: Explain how a sequence-to-sequence model uses an encoder and decoder for tasks such as translation. Describe what the decoder uses to predict target tokens during training.
  Why it matters: This explains how a language model can be conditioned on a source sequence to produce a corresponding output sequence.
• Token-by-token inference process: Trace how the decoder generates a target sequence at inference time, from its initial token through its stopping point.
  Why it matters: It distinguishes generating an output from scratch from training with an available target sequence.

Key terms: sequence-to-sequence modeling, seq2seq, source sequence, target sequence, encoder, decoder, inference, encoded source sequence, seed token, stop token

15.2.1 English-to-Spanish translation

This section prepares English–Spanish sentence pairs as tokenized, padded training data for a translation model.

• Translation pair preparation: Describe how the translation examples are organized and prepared before tokenization: each line pairs an English sentence with its Spanish translation, and the pairs are shuffled and split into training, validation, and test sets.
  Why it matters: Keeping the two sentences paired preserves the translation examples through preparation and splitting.
• Language-specific text vectorization: Explain why the section uses separate English and Spanish TextVectorization layers and customizes preprocessing to preserve the Spanish start and end markers while accounting for Spanish punctuation.
  Why it matters: The preprocessing must retain meaningful translation markers while treating the two languages’ text appropriately.
• Shifted targets and padding weights: Describe the tf.data pipeline’s inputs, target, and sample weights: it supplies tokenized English and Spanish inputs, predicts the Spanish sequence one step ahead, and excludes zero-padded labels from loss and metrics.
  Why it matters: The shifted labels define what the model predicts, while the weights prevent padding from affecting its reported loss and metrics.

Key terms: text_pairs, training, validation, test sets, TextVectorization, [start], [end], ¿, punctuation, tf.data pipeline

15.2.2 Sequence-to-sequence learning with RNNs

This section builds an RNN encoder–decoder translation model, explains how it generates translations, and identifies its evaluation and sequence-length limitations.

• Naive RNN translation limitation: Explain why predicting each target token directly from the RNN output at the corresponding source step can fail when translation changes word order.
  Why it matters: It explains why the translator needs access to the whole source sentence before generating its first word.
• Encoder decoder translation flow: Trace how the encoder represents the full source sentence and how the decoder uses that representation to predict Spanish tokens.
  Why it matters: This is the section’s central solution to translating when source and target word orders differ.
• Encoder and decoder directionality: Distinguish why a bidirectional RNN is useful for this encoder but inappropriate for this decoder.
  Why it matters: The two sides have different access to future tokens, which determines whether bidirectionality is valid.
• Translation evaluation caveat: Explain why validation next-token accuracy is only a crude measure of translation quality and describe the alternative metric mentioned.
  Why it matters: A strong next-word score does not necessarily mean the model produces good complete translations.
• Autoregressive translation inference: Trace how the trained model generates a translation and explain the inefficiency identified in the example.
  Why it matters: Generation must use the model’s own predictions, and retaining unchanged state avoids needless repeated work.
• RNN translation bottlenecks: Explain the two fundamental limitations of the RNN sequence-to-sequence approach identified here.
  Why it matters: These limits explain why the toy model struggles and motivate looking beyond this RNN design.

Key terms: RNN, source sequence, target sequence, encoder RNN, decoder RNN, initial state, Bidirectional layer, encoder, decoder, GRU

15.3 The Transformer architecture

This section introduces attention as a contextual alternative to passing sequence information through an RNN’s state and explains the Einsum notation used in Transformer code.

• RNN long-sequence limitation: Explain why RNNs can struggle to use information from much earlier in a long sequence.
  Why it matters: This limitation explains why attention is useful for sequence modeling.
• Contextual attention to sequence parts: Describe the basic idea of attention and how it differs from relying solely on recurrence to pass information across a sequence.
  Why it matters: It is the central idea behind the Transformer architecture introduced here.
• Einsum axis notation rules: Explain how an Einsum equation names input and output axes and what happens to repeated or omitted axes.
  Why it matters: These rules make the array operations appearing in Transformer code readable.

Key terms: RNN, internal state, sequence, attention, Transformer architecture, recurrent layers, Einsum notation, Einstein summation notation, axes

15.3.1 Dot-product attention

This section builds dot-product attention from relevance scores and weighted source vectors, then explains learned query-key-value projections, score scaling, and multiple attention heads.

• Selective source information: Explain how attention uses the current target token to select information from the source sequence. Describe why scores are passed through softmax before taking a weighted sum of source vectors.
  Why it matters: This is how the model can pull relevant information directly from anywhere in the source sequence.
• Dot-product relevance scores: Describe how dot products supply attention scores and how those scores are organized when both source and target contain multiple tokens.
  Why it matters: The score matrix extends the single-token attention idea to an entire target sequence.
• Learned query key value roles: Explain why attention projects its inputs and distinguish queries, keys, and values in the resulting lookup.
  Why it matters: These roles explain how the model learns what to match and what information to retrieve.
• Dot-product score scaling: Explain why the mechanism scales dot-product scores and state the scaling described here.
  Why it matters: Scaling helps keep the softmax computation stable as vector size changes.
• Multiple attention heads: Explain what problem multiple attention heads address and how their differing parameters can help.
  Why it matters: Multiple heads let attention preserve and use different kinds of relevant source information instead of relying on one blended result.

Key terms: attention scores, source sequence, target token, weighted sum, softmax, dot-product, matrix, target sequence, Dense layers, query

15.3.2 Transformer encoder block

The section explains how a Transformer encoder uses self-attention and a feedforward network to build contextual token representations, with normalization, padding masks, and stackable blocks.

• Self-attention and word context: Explain how self-attention lets each token use other tokens in the same sequence to form a context-dependent representation, as illustrated by “station” attending to “train.”
  Why it matters: This is how the encoder makes a word’s representation depend on the sentence it appears in.
• Feedforward network after attention: Justify adding two dense layers with an activation between them to an encoder block, and distinguish their role from attention’s role.
  Why it matters: Attention alone does not provide the nonlinearity the section identifies as necessary for an expressive encoder.
• Sequence-wise layer normalization: Distinguish LayerNormalization from BatchNormalization in terms of where each obtains its statistics and explain why the encoder uses LayerNormalization for sequences.
  Why it matters: It explains why the encoder uses a different normalization layer from the image models.
• Padding exclusion with attention masks: Describe how the encoder uses a source mask as an attention mask to keep tokens from attending to padding.
  Why it matters: It keeps meaningless padding tokens out of the context used to represent real tokens.
• Stackable encoder block outputs: Explain why encoder blocks can be stacked and what stacking is intended to do to the English input representation.
  Why it matters: The matching shapes explain how the encoder can deepen its representation without changing the sequence interface between blocks.

Key terms: self-attention, token, attention score, representation, MultiHeadAttention, feedforward network, dense layers, activation function, LayerNormalization, BatchNormalization

15.3.3 Transformer decoder block

This section explains how a Transformer decoder combines target-sequence self-attention with attention to encoder output, and why its self-attention needs a causal mask.

• Decoder attention roles: Explain how the decoder uses self-attention to share information among target positions and cross-attention to bring in information from the encoder output sequence.
  Why it matters: The two attention layers give the decoder access to both the target sequence so far and the encoded source.
• Source padding attention mask: Describe how the decoder uses the source mask in cross-attention and what its values mark.
  Why it matters: The source mask identifies which source positions contain tokens rather than padding.
• Causal target attention mask: Explain why decoder self-attention needs a causal mask and describe which target positions a lower-triangular mask permits each position to attend to.
  Why it matters: Causal masking prevents training-time access to future target tokens that will not be available when generating a translation.

Key terms: decoder block, self-attention, cross-attention, MultiHeadAttention, encoder output sequence, source_mask, attention_mask, padding, causal attention mask, lower-triangular

15.3.4 Sequence-to-sequence learning with a Transformer

This section builds a Transformer translation model, then uses its disappointing result to show why attention alone needs positional information to handle word order.

• Transformer translation model setup: Describe how the Transformer translation model reuses the RNN translation setup but replaces GRU layers with a TransformerEncoder and TransformerDecoder, and state how its initial training result compares with the RNN model.
  Why it matters: The comparison makes the model’s missing treatment of word order visible.
• Attention and word order: Explain why the Transformer as built cannot use word order and how the section proposes making order available to it.
  Why it matters: Word order can change meaning, so translation requires a way for the Transformer to distinguish token positions.

Key terms: GRU layers, TransformerEncoder, TransformerDecoder, attention, word order, positional information, positional embedding

15.3.5 Embedding positional information

This section adds learned positional embeddings to a translation Transformer, reports the resulting gains, and explains why attention trains faster than a GRU on accelerators.

• Position-aware word embeddings: Explain how the section gives the Transformer word-order information by adding a learned position vector to each word embedding, and why it favors this over appending raw position numbers.
  Why it matters: The Transformer needs access to where words occur, not just which words occur.
• Translation results with positions: Describe what changed when the Transformer was trained with positional embeddings and how its reported result compared with the GRU model.
  Why it matters: The reported comparison shows the practical effect of adding position information in this example.
• Attention training parallelism: Explain why the Transformer trained faster than the RNN in this comparison, distinguishing the computational reason from the difference in parameter counts.
  Why it matters: The speed advantage comes from how the computation runs, not merely from using a smaller model.
• Empirical Transformer design: Explain why the section cautions against treating the presented Transformer configuration as an optimal or permanent architecture.
  Why it matters: The implementation is a successful design, not proof that every architectural choice is best.

Key terms: positional embedding, word embedding, position vector, positional information, next word, GRU, attention, looped state passing, GPU, TPU

15.4 Classification with a pretrained Transformer

The section explains why Transformers’ training speed and ability to benefit from more data helped drive a shift toward large pretrained text models.

• Loop-Free Transformer Training: Explain why Transformers can train faster than RNNs: they do not require loops during training, which is advantageous on GPUs and TPUs.
  Why it matters: Training speed helps explain why researchers could scale Transformers up.
• Scaling and Pretrained Models: Explain how Transformers’ appetite for data and successful scaling with more data, layers, and parameters contributed to a shift toward large pretrained text models.
  Why it matters: This shift explains why the chapter turns to a pretrained Transformer for text classification.

Key terms: Transformer, RNNs, GPU, TPU, data hungry, validation performance, scaling up, large pretrained models

15.4.1 Pretraining a Transformer encoder

The section explains how BERT pretrains a bidirectional Transformer encoder on masked text and why its contextual representations can be adapted to NLP tasks.

• Bidirectional encoder representations: Explain why BERT’s encoder can represent input text using context on both sides of each position, and why that makes it suited to representing text rather than generating it in a loop.
  Why it matters: The direction of attention helps explain what the pretrained encoder is useful for.
• Masked language model pretraining: Describe how BERT is pretrained by masking tokens and predicting their original values, and distinguish this from predicting the next token using only past tokens. Explain why the masking setup does not require labeled text.
  Why it matters: Masking supplies a training task from ordinary text, making large-scale encoder pretraining practical.
• Contextual pretraining and adaptation: Explain what pretraining an entire Transformer adds beyond pretrained word embeddings, and how the pretrained model was subsequently used on NLP tasks.
  Why it matters: It explains why a pretrained encoder can serve as a starting point for specific text tasks.

Key terms: BERT, bidirectional, encoder, masked language modeling, [MASK], causal language model, unsupervised, pretrained word embeddings, context, fine-tuning

15.4.2 Loading a pretrained Transformer

This section explains how to load pretrained RoBERTa assets with KerasHub and why its tokenizer and backbone are suited to downstream tasks.

• Matching pretrained model assets: Explain why using pretrained RoBERTa requires its matching tokenizer, model architecture, and weights, and describe how KerasHub's from_preset() provides those assets.
  Why it matters: Matching the pretrained setup lets you reuse what the model learned rather than merely loading its weights.
• Subword tokenization tradeoffs: Justify RoBERTa's use of subword tokenization instead of character-level or word-level tokenization for its large web-text dataset.
  Why it matters: The tokenizer keeps both sequence length and vocabulary size practical for large-scale pretraining.
• Task agnostic Transformer backbone: Describe what the loaded RoBERTa backbone produces and why a task-specific head is needed for IMDb review classification.
  Why it matters: Separating the reusable backbone from its prediction head explains how one pretrained model can support different tasks.
• RoBERTa pretraining advantage: Explain the main reason given for RoBERTa's better performance than BERT at a comparable parameter count.
  Why it matters: It explains the choice of RoBERTa as the pretrained model in this example.

Key terms: matching tokenizer, matching model architecture, pretrained weights, KerasHub, from_preset(), subword tokenizer, character-level tokenizer, word-level tokenizer, vocabulary, Embedding layer

15.4.3 Preprocessing IMDb movie reviews

This section prepares IMDb reviews for RoBERTa fine-tuning by matching its pretrained token format and discusses why pretraining data sources matter.

• RoBERTa review preprocessing: Explain how the reviews are prepared with RoBERTa’s tokenizer and StartEndPacker, and why the packed sequence should match the format used during pretraining.
  Why it matters: Using the pretrained model’s expected input format makes fine-tuning more effective.
• Pretraining data provenance: Describe the growing scale and changing provenance of Transformer pretraining data, and explain why its sources deserve attention.
  Why it matters: A model’s training data helps explain both how it performs and where its biases may come from.

Key terms: RoBERTa tokenizer, StartEndPacker, <s>, </s>, <pad>, pretraining data, data sources, tokens, biases

15.4.4 Fine-tuning a pretrained Transformer

The section adapts a pretrained RoBERTa backbone for IMDb review classification, explains how to turn its token sequence into one prediction, and reports the accuracy and cost tradeoff.

• Contextual first-token pooling: Explain why the classifier uses the first token’s representation to turn the backbone’s output sequence into one vector per review, rather than averaging or max-pooling token vectors.
  Why it matters: This explains how a sequence of token representations becomes a representation suitable for classifying a whole review.
• Binary classification adaptation: Describe how the pretrained backbone is adapted to predict a binary movie-review label and what happens next.
  Why it matters: A pretrained sequence-producing backbone needs a task-specific output before it can classify reviews.
• Accuracy and model cost: State the reported classification result and explain its tradeoff against the earlier bigram classifier.
  Why it matters: The comparison shows both the accuracy benefit and the practical cost of the larger model.

Key terms: backbone, first token’s representation, attention, pooling, classification head, Dense projection, nonlinearity, fine-tune, IMDb, 93%

15.5 What makes the Transformer effective?

This section explains Transformer effectiveness through attention-refined embedding spaces, their capacity for complex vector programs, and the possibilities and limits of interpolation.

• Shared embedding principle: Explain how Word2Vec and Transformers both turn relationships between co-occurring tokens into proximity in a vector space, despite their different capabilities.
  Why it matters: Word2Vec provides a simpler example of the embedding principle the section uses to explain Transformers.
• Attention refined embedding spaces: Describe how attention builds successive token embedding spaces and why the section calls the resulting spaces semantically continuous and interpolative.
  Why it matters: This is the section's central account of how attention shapes meaningful Transformer representations.
• Complex vector programs: Distinguish Word2Vec's simple vector relationships from the more complex semantic functions a Transformer can represent, and explain the role of greater representation power.
  Why it matters: The contrast explains why a shared embedding principle does not imply that Word2Vec and Transformers can perform the same tasks.
• Interpolative database limits: Explain the database analogy for a Transformer and how interpolation can produce both generalizations and hallucinations without guaranteeing genuinely new insights.
  Why it matters: The analogy helps explain both surprising outputs and why they should not automatically be trusted or mistaken for human-level intelligence.

Key terms: Word2Vec, embedding space, word arithmetic, attention, semantically continuous, semantically interpolative, representation power, autoregressive optimization objective, vector programs, continuous

15.6 Summary