GPT-2

As explained in Build a Large Language Model (From Scratch)

A family of transformer-based decoder language models pretrained by OpenAI.

GPT-2 is a decoder-only transformer language model that maps token sequences to next-token logits. Its pipeline combines token and positional embeddings, repeated transformer blocks, final layer normalization, and a vocabulary-sized linear output layer. section 4.1 section 4.6

The 124M configuration specifies the model’s main data flow and dimensions:

• Accepts token-index batches shaped (batch size, sequence length), then produces context representations for every input position. section 4.1
• Adds token embeddings and positional embeddings with vocabulary size 50,257, context length 1,024, and embedding dimension 768. section 4.1 section 4.6
• Passes the combined tensor through 12 repeated transformer blocks containing masked multi-head attention, feed-forward layers, dropout, and layer normalization. section 4.6
• Applies final normalization and projects each position to 50,257 logits, yielding an output shape of (batch size, sequence length, 50,257). section 4.6
• Scales the same core architecture across 124M, 355M, 774M, and 1,558M variants by increasing embedding sizes and repeated components. section 5.5

A forward pass and pretrained generation

Tokenizing “Every effort moves you” and “Every day holds a” produces a two-row batch with four token IDs per row. A randomly initialized GPTModel returns logits shaped torch.Size([2, 4, 50257]). After loading OpenAI’s 124M weights, generation with top_k=50, temperature=1.5, and 25 new tokens produces text beginning “Every effort moves you toward finding an ideal new way to practice something!” section 4.6 section 5.5

Figure 4.2 A GPT model. In addition to the embedding layers, it consists of one or more transformer blocks containing the masked multi-head attention module we previously implemented.

Use the architecture figure to trace token embeddings and positional embeddings through repeated transformer blocks to vocabulary logits. section 4.1

OpenAI’s released weights can be downloaded and assigned to the implementation after matching configuration settings and checking tensor shapes. The weights were originally saved with TensorFlow, while the model implementation uses PyTorch. section 5.5

How Build a Large Language Model (From Scratch) covers it

How deep it goes

Descends from architecture flow to PyTorch modules, tensor shapes, parameter counts, and pretrained weight assignment.

Examples

Demonstrates two four-token batches, 50,257-way logits, model dimension inspection, pretrained loading, and sampled continuation.

What it assumes

Assumes tokenization, embeddings, and masked attention, while introducing the complete transformer-stack assembly and parameter accounting.

What you can do after

Builds a GPT-2 architecture, loads OpenAI weights, and generates continuations with configurable sampling.

How it connects

Connects tokenization, embeddings, positional encoding, masked multi-head attention, transformer blocks, layer normalization, and text generation.

liveBookmanning.com

Read Build a Large Language Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionGPT-2 is an autoregressive transformer language model developed by OpenAI and trained to generate text from preceding context.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA decoder-only Transformer language model used here as the example model whose embeddings and positional encoding are being described.
• 

  liveBookmanning.com
  Sutskever's ListGPT-2 is a generative transformer language model released by OpenAI with versions ranging up to 1.5 billion parameters.
• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsA decoder-only Transformer language model used for text generation, here serving as the architecture underlying ProtGPT2.
• 

  liveBookmanning.com
  Causal AIGPT-2 is an autoregressive transformer language model developed by OpenAI for generating and completing text.

Compare how 6 books explain GPT-2 →

Related concepts

• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• decoder-only transformerkind of · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• PyTorchuses · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
• GELUuses · GELU, or Gaussian error linear unit, is a smooth neural-network activation function that scales an input by the cumulative distribution function of the standard normal distribution at that input.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of Build a Large Language Model (From Scratch). The section links point to the parts of the book it draws on.