liveBook · Manning

GPT-2

As explained in Rearchitecting LLMs

GPT2Model is a GPT-2 transformer model implementation that organizes embeddings, transformer blocks, normalization, and output projection.

GPT2Model organizes token and position embeddings, dropout, a stack of Transformer blocks, and final normalization. Its forward pass sends embeddings through the blocks in sequence, then applies final normalization. The language-modeling head is added by GPT2LMHeadModel, not GPT2Model. section 3.1 section 3.1.1

The implementation’s main stages are:

  • wte maps token IDs to token embeddings; wpe supplies position embeddings. Their combination carries semantic and positional information. section 3.1.1
  • Each GPT2Block calls attention and then its MLP; the model applies ln_f after the block stack. section 3.1.1
  • The shown DistilGPT2 configuration has vocabulary size 50,257, embedding size 768, 1,024 position embeddings, and six blocks. section 3.1
Figure 3.2 Anatomy of DistilGPT2. On the left, the general architecture is shown: data flows from the Embeddings layer, passes through the stack of six Transformer Blocks, and ends at the output layer. On the right, a "zoom" into one of those blocks reveals a LayerNorm → GPT2Attention → LayerNorm → GPT2MLP flow, where each module processes the information before passing it to the next.

Use the architecture diagram to trace the embedding-to-block flow and inspect the sequence of modules inside a Transformer block. section 3.1.1

How Rearchitecting LLMs covers it

How deep it goes
Traces constructor components and nested forward calls, while deferring detailed operations inside attention and the MLP.
Examples
Uses DistilGPT2 dimensions and a diagram showing embeddings, six blocks, and the block’s module sequence.
What it assumes
Assumes token IDs are produced by an external tokenizer and explains how the embedding layer uses them.
What you can do after
Evaluate the model’s component structure and trace its high-level data flow from embeddings through final normalization.
How it connects
Connects tokenization, embeddings, Transformer blocks, attention, MLP, layer normalization, and GPT2LMHeadModel.

Also explained in

Compare how 7 books explain GPT-2 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.