What does section 9.2, Self-supervised learning, cover?

The section builds a character-pair model for generating names and uses it to introduce learning from unlabeled data. Explain that statistical generation uses probabilities of ordered character sequences to predict and generate text, as illustrated by the letters in “John.” Trace how the section builds a bigram model from names: count consecutive character pairs in a matrix, normalize the counts into probabilities, and sample from the resulting distribution. Describe the quality of names sampled from the bigram distribution and the limitation the section observes.

What does section 9.2.1, Limits of the bigram model, cover?

The section shows why counting character-sequence probabilities becomes impractical as vocabulary or context grows, and presents neural networks as a way to generalize beyond those counts. Explain why storing a probability for every possible character pair becomes inefficient as the vocabulary grows, including what data sparsity means here. Explain how conditioning the next character on more preceding characters increases the number of probabilities the model must calculate. Describe the section’s stated reason neural networks are a better fit than separately calculating probabilities for character combinations.

What does section 9.3, Generating our training data, cover?

The section turns names into next-character training examples, then pads and batches those examples for model training. Explain how names provide their own input-target pairs: encode characters as integers, use each prefix of a name as an input, and use the following character as its target. Describe why the sequences are padded, which values pad inputs and targets, and how the prepared pairs are gathered into batches.

What does section 9.4, Embeddings and linear layers, cover?

This section builds a character-level name generator from trainable embeddings and a linear layer, then explains how its training examples, loss, and autoregressive sampling work. Explain why the model replaces integer character IDs with vectors and how nn.Embedding supplies those vectors. Trace how the new model turns an input subsequence into a next-character prediction. Describe how names supply multiple training examples and how the loss handles their outputs and invalid targets.

What does section 9.4.1, Visualizing embeddings, cover?

This section uses a 3D visualization to show how training gives character embeddings a structure that reflects relationships among characters. Describe how the example plots each character embedding and distinguish its three dimensions from the larger embeddings typically used in real-world applications. Explain what the relative positions of characters in the visualization show about the trained embeddings.

What does section 9.5, Attention, cover?

This section explains why static character embeddings need context-sensitive attention and introduces query, key, and value as the way attention selects relevant information. Explain why a fixed embedding cannot capture a character’s changing role and how attention instead gives different parts of the sequence importance according to context. Describe what queries, keys, and values represent, and how a query–key match affects the weight given to a value.

What does section 9.5.1, Dot product self-attention, cover?

This section explains how self-attention combines information across sequence positions and how query–key dot products produce the weights for that combination. Explain how a self-attention block produces an output for each sequence position by taking a weighted sum of value vectors from all positions, allowing more relevant inputs to contribute more strongly. Describe how separate linear transformations create query and key vectors, whose dot products are passed through softmax to obtain normalized attention weights. Explain how the implementation uses matrix multiplication to compute the dot products efficiently.

What does section 9.5.2, Scaled dot product causal self-attention, cover?

The section extends dot-product attention with batching, causal masking, and scaling, then applies it to character generation and interprets the resulting attention weights. Explain why the implementation adds batching, causal masking, and scaling to dot-product attention. Trace how the new model uses embeddings, attention, and linear layers, and contrast it with flattening and concatenating embeddings. Describe what the returned attention weights reveal in the visualization of the name “john.”

What does section 9.6, Transformers, cover?

Introduces attention as the core of transformers, outlines the original encoder–decoder architecture, and explains the prominence of decoder-only models. Explain that attention is the central mechanism of the Transformer architecture and connects it to the attention mechanism implemented in the preceding section. Describe how the original transformer uses an encoder to turn an input sequence into representations of relationships among its elements, then a decoder to generate an output sequence from those representations. Explain why decoder-only transformers receive emphasis here: they power leading large language models and are widely used for text generation.

What does section 9.6.1, The decoder, cover?

This section builds a GPT-style decoder from transformer blocks, explains the components added to the earlier attention-MLP block, and reports how the resulting name generator performs. Describe how the decoder adds positional information to input embeddings, passes them through multiple transformer blocks, and uses a final linear layer to produce output for sequence generation. Explain why residual connections are added to the decoder block and how they help gradients pass through a deep network. Explain how multiheaded attention uses separate learned query, key, and value projections in parallel heads, then combines their outputs.

What does section 9.7.1, The encoder, cover?

This section distinguishes the transformer encoder from the decoder and describes what its output is used for. Explain how an encoder differs from a decoder in what tokens it can attend to and describe its output. Describe why an encoder’s output is useful for text classification, and identify BERT as an example encoder model.

What should I know before reading chapter 9?

Distinguishing birds from airplanes (chapter 7): The earlier classification example provides a contrast with generation.; Converting text to numbers (chapter 4): Representing text as numbers.; Training the classifier (chapter 7): Earlier training of a classifier with specified targets.; Text embeddings (chapter 4): The earlier introduction to text embeddings.

Chapter 9: How transformers work

What you will learn in chapter 9 of Deep Learning with PyTorch, Second Edition

The introduction contrasts generative modeling with earlier prediction tasks and uses ELIZA to show why realistic text generation requires more than fixed response rules.

After this chapter you will be able to

Generation and hidden structure. Explain why generation differs from regression and classification, and how learning hidden patterns in existing data makes generating new material possible.
ELIZA and conversational context. Explain how ELIZA produced plausible conversational replies and why those replies did not demonstrate understanding of the conversation.
Character Vocabulary and Name Boundaries. Explain how the name-generation vocabulary represents lowercase letters as integer tokens and uses a special token to mark the start and end of a name.
Limits of Random Letter Selection. Explain why choosing letters with equal probability produces poor names and why letter-to-letter patterns matter for the stated task.
Bigram Matrix Construction. Trace how the section builds a bigram model from names: count consecutive character pairs in a matrix, normalize the counts into probabilities, and sample from the resulting distribution.
Self-Supervision Versus Labels. Distinguish supervised learning with explicit, often manually labeled targets from self-supervised learning that infers what to learn from unlabeled data, using the name example.
Vocabulary growth and data sparsity. Explain why storing a probability for every possible character pair becomes inefficient as the vocabulary grows, including what data sparsity means here.
Longer context probability growth. Explain how conditioning the next character on more preceding characters increases the number of probabilities the model must calculate.
Neural representations as an alternative. Describe the section’s stated reason neural networks are a better fit than separately calculating probabilities for character combinations.
Next-character training pairs. Explain how names provide their own input-target pairs: encode characters as integers, use each prefix of a name as an input, and use the following character as its target.
Trainable character embeddings. Explain why the model replaces integer character IDs with vectors and how nn.Embedding supplies those vectors.
Next-character model path. Trace how the new model turns an input subsequence into a next-character prediction.
Training pairs and loss. Describe how names supply multiple training examples and how the loss handles their outputs and invalid targets.
Autoregressive name generation. Trace how the trained model generates a name and state what improved about its outputs.
Learned character relationships. Explain what the relative positions of characters in the visualization show about the trained embeddings.
Context-sensitive character attention. Explain why a fixed embedding cannot capture a character’s changing role and how attention instead gives different parts of the sequence importance according to context.
Query key value roles. Describe what queries, keys, and values represent, and how a query–key match affects the weight given to a value.
Weighted information across positions. Explain how a self-attention block produces an output for each sequence position by taking a weighted sum of value vectors from all positions, allowing more relevant inputs to contribute more strongly.
Query key attention weights. Describe how separate linear transformations create query and key vectors, whose dot products are passed through softmax to obtain normalized attention weights. Explain how the implementation uses matrix multiplication to compute the dot products efficiently.
Three attention extensions. Explain why the implementation adds batching, causal masking, and scaling to dot-product attention.
Attention based character model. Trace how the new model uses embeddings, attention, and linear layers, and contrast it with flattening and concatenating embeddings.
Attention at the transformer core. Explain that attention is the central mechanism of the Transformer architecture and connects it to the attention mechanism implemented in the preceding section.
Original encoder and decoder roles. Describe how the original transformer uses an encoder to turn an input sequence into representations of relationships among its elements, then a decoder to generate an output sequence from those representations.
GPT-style decoder architecture. Describe how the decoder adds positional information to input embeddings, passes them through multiple transformer blocks, and uses a final linear layer to produce output for sequence generation.
Residual gradient paths. Explain why residual connections are added to the decoder block and how they help gradients pass through a deep network.
Parallel attention heads. Explain how multiheaded attention uses separate learned query, key, and value projections in parallel heads, then combines their outputs.
Token position information. Justify adding positional embeddings to token embeddings in a transformer decoder.
Unmasked encoder attention. Explain how an encoder differs from a decoder in what tokens it can attend to and describe its output.
Encoder and decoder roles. Describe how the encoder turns an input sequence into representations of relationships among its elements, which the decoder uses to generate an output sequence.
Decoder attention to encoder output. Explain how the encoder output supplies the keys and values for decoder-encoder attention, allowing the decoder to use input information when generating an output sequence. Relate this to translation between languages with different word orders or grammatical structures.
Text to numeric tokens. Explain that a tokenizer prepares raw text for a model by splitting it into tokens and converting those tokens into numbers.
Token-based decoder flexibility. Explain why the decoder architecture can be trained on the new dataset without changing it for word tokens.
Image patches as tokens. Describe how ViT turns an image into a sequence of patch tokens, embeds them, adds positional information, and passes them through transformer encoder blocks.
Classification class token. Explain why ViT prepends a learnable [CLS] token and uses its output, rather than the other token outputs, for image classification.
Transformer capabilities across content. Describe how the chapter progresses from generating character sequences to producing longer text and understanding images, and explain that the Transformer architecture has made human-like text generation a reality.

Cover of Deep Learning with PyTorch, Second Edition

liveBook manning.com

Read chapter 9 on liveBook Get the book at manning.com

What it builds on

Distinguishing birds from airplanes (chapter 7) · The earlier classification example provides a contrast with generation.
Converting text to numbers (chapter 4) · Representing text as numbers.
Training the classifier (chapter 7) · Earlier training of a classifier with specified targets.
Text embeddings (chapter 4) · The earlier introduction to text embeddings.

Section by section

Introduction

The introduction contrasts generative modeling with earlier prediction tasks and uses ELIZA to show why realistic text generation requires more than fixed response rules.

Generation and hidden structure: Explain why generation differs from regression and classification, and how learning hidden patterns in existing data makes generating new material possible.
Why it matters: This explains what a generative model must learn before it can produce something new.
ELIZA and conversational context: Explain how ELIZA produced plausible conversational replies and why those replies did not demonstrate understanding of the conversation.
Why it matters: ELIZA shows why a convincing individual reply is not the same as following a conversation.

Key terms: generation, latent structure, implicit rules, probabilistic model, ELIZA, pattern matching, rule-based statements, conversational context

9.1 A motivating example: Generating names character by character

The section introduces character-by-character name generation, shows how to represent names as token sequences, and explains why random letter selection is a poor starting point.

Character Vocabulary and Name Boundaries: Explain how the name-generation vocabulary represents lowercase letters as integer tokens and uses a special token to mark the start and end of a name.
Why it matters: This representation turns names into sequences a language model can work with.
Limits of Random Letter Selection: Explain why choosing letters with equal probability produces poor names and why letter-to-letter patterns matter for the stated task.
Why it matters: It identifies what a better name generator must capture to produce natural-sounding names.

Key terms: vocabulary, token, sequence, $, uniform distribution, probability, generative language models

9.2 Self-supervised learning

The section builds a character-pair model for generating names and uses it to introduce learning from unlabeled data.

Statistical Text Generation: Explain that statistical generation uses probabilities of ordered character sequences to predict and generate text, as illustrated by the letters in “John.”
Why it matters: This gives you the basic idea behind generating a name from learned letter patterns.
Bigram Matrix Construction: Trace how the section builds a bigram model from names: count consecutive character pairs in a matrix, normalize the counts into probabilities, and sample from the resulting distribution.
Why it matters: The matrix makes the section’s first learned generator concrete.
Bigram Generation Quality: Describe the quality of names sampled from the bigram distribution and the limitation the section observes.
Why it matters: Readable output is not the same as convincing output.
Self-Supervision Versus Labels: Distinguish supervised learning with explicit, often manually labeled targets from self-supervised learning that infers what to learn from unlabeled data, using the name example.
Why it matters: It explains how the names themselves can provide training information without someone labeling each example.

Key terms: statistical generation, probability distributions, text sequences, bigram model, character pairs, bigram matrix, sample, realistic, supervised learning, self-supervised learning

9.2.1 Limits of the bigram model

The section shows why counting character-sequence probabilities becomes impractical as vocabulary or context grows, and presents neural networks as a way to generalize beyond those counts.

Vocabulary growth and data sparsity: Explain why storing a probability for every possible character pair becomes inefficient as the vocabulary grows, including what data sparsity means here.
Why it matters: It explains why a table of observed pairs becomes a poor approach for larger vocabularies.
Longer context probability growth: Explain how conditioning the next character on more preceding characters increases the number of probabilities the model must calculate.
Why it matters: It shows why simply extending the probability table is not a practical way to capture longer-range dependencies.
Neural representations as an alternative: Describe the section’s stated reason neural networks are a better fit than separately calculating probabilities for character combinations.
Why it matters: It connects the probability-table limitations to the book’s reason for turning to neural networks.

Key terms: bigram model, character pairs, data sparsity, vocabulary, longer-range dependencies, probability of the next character, dense, continuous representations, generalize to unseen patterns, fixed parameter count

9.3 Generating our training data

The section turns names into next-character training examples, then pads and batches those examples for model training.

Next-character training pairs: Explain how names provide their own input-target pairs: encode characters as integers, use each prefix of a name as an input, and use the following character as its target.
Why it matters: This explains where next-character targets come from when the names have no separately supplied labels.
Padding and training batches: Describe why the sequences are padded, which values pad inputs and targets, and how the prepared pairs are gathered into batches.
Why it matters: Uniform lengths let the examples be processed together without treating padded targets as characters to predict.

Key terms: input-target pairs, encode, decode, input sequence, target sequence, pad, input sequences, target sequences, batch size, loss function

9.4 Embeddings and linear layers

This section builds a character-level name generator from trainable embeddings and a linear layer, then explains how its training examples, loss, and autoregressive sampling work.

Trainable character embeddings: Explain why the model replaces integer character IDs with vectors and how nn.Embedding supplies those vectors.
Why it matters: This explains how character IDs become representations the model can learn from.
Next-character model path: Trace how the new model turns an input subsequence into a next-character prediction.
Why it matters: The path from subsequence to prediction is the architecture being trained in this section.
Training pairs and loss: Describe how names supply multiple training examples and how the loss handles their outputs and invalid targets.
Why it matters: The loss must account for every valid next-character example produced from the names.
Autoregressive name generation: Trace how the trained model generates a name and state what improved about its outputs.
Why it matters: Feeding predictions back in lets a next-character model produce a complete name.

Key terms: text embeddings, nn.Embedding, vocabulary size, trainable lookup table, backpropagation, subsequence, embeddings, flatten, linear layer, next character

9.4.1 Visualizing embeddings

This section uses a 3D visualization to show how training gives character embeddings a structure that reflects relationships among characters.

Three-dimensional embedding visualization: Describe how the example plots each character embedding and distinguish its three dimensions from the larger embeddings typically used in real-world applications.
Why it matters: The plot is an accessible picture of embeddings, not a limit on how many dimensions they can have.
Learned character relationships: Explain what the relative positions of characters in the visualization show about the trained embeddings.
Why it matters: The arrangement makes the effects of training visible rather than leaving embeddings as unexplained numbers.

Key terms: embedding, tensor of size 3, 3D space, embeddings, vowels, consonants, special character $

9.5 Attention

This section explains why static character embeddings need context-sensitive attention and introduces query, key, and value as the way attention selects relevant information.

Context-sensitive character attention: Explain why a fixed embedding cannot capture a character’s changing role and how attention instead gives different parts of the sequence importance according to context.
Why it matters: This is why the model needs more than character embeddings to generate contextually appropriate names.
Query key value roles: Describe what queries, keys, and values represent, and how a query–key match affects the weight given to a value.
Why it matters: These roles explain how attention selects information relevant to a position in the sequence.

Key terms: embeddings, attention mechanism, context, query, key, value

9.5.1 Dot product self-attention

This section explains how self-attention combines information across sequence positions and how query–key dot products produce the weights for that combination.

Weighted information across positions: Explain how a self-attention block produces an output for each sequence position by taking a weighted sum of value vectors from all positions, allowing more relevant inputs to contribute more strongly.
Why it matters: It explains how a position can draw on relevant information elsewhere in the sequence.
Query key attention weights: Describe how separate linear transformations create query and key vectors, whose dot products are passed through softmax to obtain normalized attention weights. Explain how the implementation uses matrix multiplication to compute the dot products efficiently.
Why it matters: It connects the weights used to mix information with a computation the model can perform efficiently.

Key terms: self-attention, value vectors, attention weights, weighted sum, query vectors, key vectors, dot product, softmax, matrix multiplication

9.5.2 Scaled dot product causal self-attention

The section extends dot-product attention with batching, causal masking, and scaling, then applies it to character generation and interprets the resulting attention weights.

Three attention extensions: Explain why the implementation adds batching, causal masking, and scaling to dot-product attention.
Why it matters: These extensions make attention suitable for batched, next-character prediction.
Attention based character model: Trace how the new model uses embeddings, attention, and linear layers, and contrast it with flattening and concatenating embeddings.
Why it matters: It shows what attention contributes to the character generator, even though its performance is similar to the previous model’s.
Character attention weight interpretation: Describe what the returned attention weights reveal in the visualization of the name “john.”
Why it matters: The visualization makes the model’s use of character context observable.
Built in attention function: Distinguish the section’s attention implementation from PyTorch’s built-in function in terms of results and production use.
Why it matters: Working through the mechanism aids understanding, while the optimized function is the practical production choice.

Key terms: batching, causal masking, scaling, autoregressive models, embedding layer, attention mechanism, linear layers, flattening, attention weights, attention score

9.6 Transformers

Introduces attention as the core of transformers, outlines the original encoder–decoder architecture, and explains the prominence of decoder-only models.

Attention at the transformer core: Explain that attention is the central mechanism of the Transformer architecture and connects it to the attention mechanism implemented in the preceding section.
Why it matters: This connects the attention mechanism you just studied to the larger Transformer architecture.
Original encoder and decoder roles: Describe how the original transformer uses an encoder to turn an input sequence into representations of relationships among its elements, then a decoder to generate an output sequence from those representations.
Why it matters: The two roles provide a basic map of the original architecture before examining its components separately.
Decoder-only model prominence: Explain why decoder-only transformers receive emphasis here: they power leading large language models and are widely used for text generation.
Why it matters: This explains the practical relevance of focusing on decoder-only transformers.

Key terms: Transformer architecture, attention mechanism, encoder, decoder, input sequence, representations, output sequence, decoder-only transformers, large language models, text generation

9.6.1 The decoder

This section builds a GPT-style decoder from transformer blocks, explains the components added to the earlier attention-MLP block, and reports how the resulting name generator performs.

GPT-style decoder architecture: Describe how the decoder adds positional information to input embeddings, passes them through multiple transformer blocks, and uses a final linear layer to produce output for sequence generation.
Why it matters: This gives you the overall path from input embeddings to generated output.
Residual gradient paths: Explain why residual connections are added to the decoder block and how they help gradients pass through a deep network.
Why it matters: Skip paths help explain how the decoder can be built from multiple blocks.
Parallel attention heads: Explain how multiheaded attention uses separate learned query, key, and value projections in parallel heads, then combines their outputs.
Why it matters: Parallel heads let one decoder block represent diverse relationships at once.
Token position information: Justify adding positional embeddings to token embeddings in a transformer decoder.
Why it matters: Without position information, the decoder would lose distinctions that depend on word order.
Layer normalization distinction: Distinguish layer normalization from batch normalization by what each normalizes across, and state its purpose here.
Why it matters: The normalization axis is the key distinction between these two components.
Decoder training outcome: Describe the reported tradeoff and generation results for the implemented name-generating decoder.
Why it matters: The results connect the added architectural complexity to the example's observed performance.

Key terms: decoder-only transformers, input embeddings, positional encoding, transformer blocks, linear layer, residual connections, skip connections, vanishing gradients, multiheaded attention, attention heads

9.7 Other Transformer architectures

9.7.1 The encoder

This section distinguishes the transformer encoder from the decoder and describes what its output is used for.

Unmasked encoder attention: Explain how an encoder differs from a decoder in what tokens it can attend to and describe its output.
Why it matters: The absence of a causal mask explains why the encoder can represent the whole input sequence.
Encoder classification uses: Describe why an encoder’s output is useful for text classification, and identify BERT as an example encoder model.
Why it matters: These examples connect the encoder’s sequence representation to practical classification tasks.

Key terms: transformer encoder, causal masking, fixed-size vector representation, text classification, sentiment analysis, spam detection, topic classification, BERT, pretrained

9.7.2 The encoder-decoder

This section explains how an encoder and decoder form a transformer that generates an output sequence from an input sequence.

Encoder and decoder roles: Describe how the encoder turns an input sequence into representations of relationships among its elements, which the decoder uses to generate an output sequence.
Why it matters: The two roles explain how the architecture connects an input sequence to a generated output sequence.
Decoder attention to encoder output: Explain how the encoder output supplies the keys and values for decoder-encoder attention, allowing the decoder to use input information when generating an output sequence. Relate this to translation between languages with different word orders or grammatical structures.
Why it matters: This connection explains how the decoder can draw on the input sentence when producing a translation.

Key terms: encoder, decoder, input sequence, output sequence, representations, encoder output, keys, values, decoder-encoder attention, machine translation

9.8 Tokenization

The section explains why text models use tokenizers and why practical tokenizers often use words or subwords rather than individual characters.

Text to numeric tokens: Explain that a tokenizer prepares raw text for a model by splitting it into tokens and converting those tokens into numbers.
Why it matters: Tokenization is the bridge between readable text and the numeric input a model can process.
Word and subword tokens: Justify using words or subwords instead of only characters as tokens in the example discussed here.
Why it matters: The choice of token size affects whether a token represents a meaningful piece of text.

Key terms: tokenizer, raw text, tokens, words, subwords, characters

9.8.1 Generating sentences

The section reuses a token-based decoder to generate sentences and adjusts how its token output is displayed.

Token-based decoder flexibility: Explain why the decoder architecture can be trained on the new dataset without changing it for word tokens.
Why it matters: Thinking in tokens explains why the decoder can be reused for sentence generation.
Readable generated sentence formatting: Describe why the sample-generation output needs a small formatting change.
Why it matters: Formatting lets a reader inspect generated sentences rather than a raw token list.

Key terms: tokens, decoder, model architecture, generate_samples, format_sequence, tokenizer

9.9 The Vision Transformer

This section explains how a Vision Transformer treats image patches as tokens, uses an encoder to process them, and uses the resulting representations for vision tasks.

Image patches as tokens: Describe how ViT turns an image into a sequence of patch tokens, embeds them, adds positional information, and passes them through transformer encoder blocks.
Why it matters: It connects the Transformer concepts used for text to the image input ViT actually processes.
Classification class token: Explain why ViT prepends a learnable [CLS] token and uses its output, rather than the other token outputs, for image classification.
Why it matters: It explains how a sequence of patch representations yields one image-level classification.
Transformers beyond text: Explain how ViT illustrates that Transformers are not limited to text, and distinguish classification's use of the [CLS] output from other vision tasks' possible use of all encoder outputs.
Why it matters: The patch-token example shows why the same architecture can serve tasks beyond text processing.

Key terms: Vision Transformer, patches, tokens, positional embeddings, transformer encoder blocks, [CLS] token, self-attention, classification, Transformer architecture, attention

9.10 Conclusion

The conclusion summarizes the capabilities demonstrated in the chapter and urges a careful understanding of what transformers can and cannot be said to do.

Transformer capabilities across content: Describe how the chapter progresses from generating character sequences to producing longer text and understanding images, and explain that the Transformer architecture has made human-like text generation a reality.
Why it matters: This puts the chapter’s examples in perspective as applications of a consequential architecture.
Apparent thinking and practitioner judgment: Distinguish systems that appear to think autonomously from a claim that they actually do, and state why the conclusion urges practitioners to understand the technology’s mechanics.
Why it matters: Understanding how the systems work helps practitioners interpret their striking behavior without assuming it proves autonomous thought.

Key terms: Transformer architecture, natural language generation, content understanding, appear to think autonomously, mechanics

9.11 Exercises

9.12 Summary

Where this is used later

SAM architecture (chapter 15) · Introduces the Vision Transformer used by the image encoder.

Concepts explained in this chapter

transformer architectureA neural network architecture for sequence modeling and content understanding based on attention rather than recurrence or convolution.
generationThe task of producing a token sequence as output, conditioned on preceding input or context.
attentionA neural-network technique that helps a model focus on the most relevant parts of its input when computing representations.
embeddingA one-dimensional numeric data structure used here as a mathematical object on which sums and dot products can be computed.
decodeA tokenizer function that converts numeric token representations back into text.
linear layerA PyTorch linear layer module that applies an affine transformation to map features to outputs.
keyThe attention component that represents what a position contains and can be matched against a query.
queryA learned projection of an input position used to score how relevant other positions are in attention.
valueThe transformed representation vectors that are combined using attention weights to produce the output for each position.
GPT modelA decoder-only transformer architecture used for autoregressive text generation without a separate encoder stack.
tokenA text unit produced by tokenization, which may be a character, word, or subword depending on the tokenizer design.
positional embeddingAn embedding that injects token-order information into transformer inputs because the architecture is not inherently sequential.
bigram modelA sequence model that predicts the next symbol from the immediately preceding symbol using pairwise transition probabilities.
statistical generationA probabilistic model that generates text by learning sequence distributions from data and sampling from them.
multi-head attentionAn attention mechanism that runs several attention heads in parallel, each learning different projections and relationships.
Vision TransformerA transformer-based neural network adapted for computer vision tasks, especially image classification, by processing image patches with attention.
softmaxA normalization function that converts a vector of scores into a probability distribution by exponentiating and renormalizing the values.
residual connectionA connection that bypasses one or more network layers to provide an alternate path for activations or gradients.
encodingA tokenizer function that converts text into numeric token representations.
labelA target label restricted to one of two classes, used here in binary classification and metric computation.
vocabularyThe set of distinct symbols or words a language model can assign probabilities to in this character- or word-based setting.
training datasetThe subset of data used to fit a model.
fine-tuningThe process of adapting a pretrained model to a new task by continuing training on task-specific data.
classifierA supervised learning task that assigns a text sequence to one of several categories.
decoder-only transformerA transformer variant that uses only the decoding stack to generate outputs autoregressively from prior context.
batch sizeThe number of training examples processed together in one update step.
Hugging FaceA repository service for discovering, sharing, and exchanging pretrained models and weights.
datasetA collection of examples and annotations used for model training or evaluation.
language modelA language model with enough scale to generate and answer with fluent text from prompts.
normalizationNormalization is the process of transforming a value, representation, or data structure to satisfy a specified standard, scale, or structural constraint.
foundational modelA neural network already trained on prior data and reused for inference or further training.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.