GPT-2XL is a large decoder-only Transformer used to connect architecture choices with model size and parameter count. Its input token indexes become word and positional embeddings, pass through 48 decoder blocks, and produce logits for 50,257 vocabulary tokens. section 11.2.4
A GPT-2XL decoder block combines:
- Causal self-attention with layer normalization and a residual connection. section 11.2.4
- A GELU feed-forward network with layer normalization and a residual connection. section 11.2.4
- A 1,600-dimensional representation, expanded to 6,400 dimensions inside the feed-forward network. section 11.2.4
- A final layer normalization and linear language-model head that maps representations to vocabulary logits. section 11.2.4
Counting the model
Instantiating GPT2XL and summing parameters in its transformer body produces approximately 1,557.61 million parameters, or more than 1.5 billion. This count excludes the final linear head.
The architecture can be matched to pretrained OpenAI GPT-2XL weights through Hugging Face Transformers. The checkpoint represents several attention and feed-forward projections with Conv1d modules, whereas the reconstructed model uses linear layers; the corresponding weight matrices therefore require transposition during loading. section 11.3.1
