GPT-2

As explained in Learn Generative AI with PyTorch

A decoder-only Transformer language model used here as the example model whose embeddings and positional encoding are being described.

GPT-2 is a decoder-only Transformer that generates text by predicting the next token from preceding tokens. Unlike an encoder–decoder translator, it has no encoder stack or encoder-derived inputs. Its decoder blocks use causal self-attention so generation depends on earlier tokens in the sequence. section 11.1 section 11.1.1

Figure 11.1 The architecture of the GPT-2 model. GPT-2 is a decoder-only Transformer, consisting of N identical decoder layers. Each decoder block contains two sublayers. The first sublayer is a causal self-attention layer. The second is a feed-forward network. Each sublayer uses layer normalization and a residual connection. The input is first passed through word embedding and positional encoding, and the sum is then passed through the decoder. The output from the decoder goes through layer normalization and a linear layer.

Trace token and position embeddings through repeated decoder blocks, then through the final normalization and vocabulary-output layer. section 11.1.1

A practical build follows the path from text to a next-token prediction:

• Tokenize text with byte pair encoding (BPE), then turn token indexes into learned word embeddings. For “this is a prompt,” four tokens become a 4 × 1,600 embedding matrix in GPT-2XL. section 11.1.2
• Add learned positional embeddings so the model can distinguish token order. GPT-2 supports positions in sequences of up to 1,024 tokens. section 11.1.2
• Pass the combined embeddings through decoder blocks. Each block pairs causal self-attention with a position-wise feed-forward network and uses layer normalization and residual connections. section 11.1.1
• Apply final layer normalization and a linear layer to produce an output for each of the 50,257 vocabulary tokens. section 11.1.1

Model size changes the build: GPT-2 variants have 12, 24, 36, or 48 decoder blocks. GPT-2XL uses 48 blocks, 1,600-dimensional embeddings, and about 1.5 billion parameters. Building that architecture does not train it; load released pretrained weights before using it to generate meaningful text. section 11.1.1 section 11.3

A smaller model for training practice

A simplified, 5.12-million-parameter GPT-2-style model can instead be trained on three novels. Generating from “the old man saw the shark near the” makes a useful check: compare coherence with passages copied from training data. In this example, the 20-epoch model is less coherent than the 40-epoch model but has a lower risk of copying. section 12.4 section 12.4.3

How Learn Generative AI with PyTorch covers it

How deep it goes

Works through embeddings, decoder blocks, and model assembly, but loads pretrained weights rather than training the full GPT-2XL model.

Examples

Uses a four-token prompt to illustrate embedding shapes and a shark-related prompt to compare generated passages.

What it assumes

Assumes earlier exposure to Transformer translation and embeddings while introducing BPE and GELU during the build.

What you can do after

Build it from scratch as GPT-2XL, load released weights to generate text, or train a smaller GPT-style model.

How it connects

Connects byte pair encoding and positional embeddings to causal self-attention, feed-forward networks, and next-token prediction.

liveBookmanning.com

Read Learn Generative AI with PyTorch on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionGPT-2 is an autoregressive transformer language model developed by OpenAI and trained to generate text from preceding context.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A family of transformer-based decoder language models pretrained by OpenAI.
• 

  liveBookmanning.com
  Sutskever's ListGPT-2 is a generative transformer language model released by OpenAI with versions ranging up to 1.5 billion parameters.
• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsA decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.
• 

  liveBookmanning.com
  Causal AIGPT-2 is an autoregressive transformer language model developed by OpenAI for generating and completing text.

Compare how 6 books explain GPT-2 →

Related concepts

• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• decoder-only transformerkind of · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
• GELUuses · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Learn Generative AI with PyTorch. The section links point to the parts of the book it draws on.