GPT-2XL is a large GPT-style decoder-only Transformer used to compare architecture scale and parameter count. It processes token indexes through word and positional embeddings, 48 decoder blocks, final layer normalization, and a vocabulary-sized linear head. section 11.2.4
Its main construction choices are:
- Stack 48 decoder layers, each combining causal self-attention and a GELU feed-forward network with layer normalization and residual connections. section 11.2.4
- Produce logits for all 50,257 vocabulary tokens; apply softmax later when converting logits into generation probabilities. section 11.2.4
The architecture can be recreated from scratch and then aligned with pretrained OpenAI GPT-2XL weights. Hugging Face represents several projection layers with Conv1d modules, while the recreated model uses linear layers, so selected weight matrices must be transposed during loading. GPT-2XL's own future-token masking parameters are omitted because causal masking is implemented separately. section 11.3.1
Inspecting the model scale
Instantiate GPT2XL(config), sum the parameters in model.transformer, and print the result as millions: the reported count is 1557.61M.
Listing 11.6 Building the GPT-2XL model
class GPT2XL(nn.Module):
def __init__(self, config):
super().__init__()
self.block_size = config.block_size
self.transformer = nn.ModuleDict(dict(
wte = nn.Embedding(config.vocab_size, config.n_embd),
wpe = nn.Embedding(config.block_size, config.n_embd),
drop = nn.Dropout(config.embd_pdrop),
h = nn.ModuleList([Block(config)
for _ in range(config.n_layer)]),
ln_f = nn.LayerNorm(config.n_embd),))
self.lm_head = nn.Linear(config.n_embd,
config.vocab_size, bias=False)
def forward(self, idx, targets=None):
b, t = idx.size()
pos = torch.arange(0,t,dtype=torch.long).unsqueeze(0)
tok_emb = self.transformer.wte(idx)
pos_emb = self.transformer.wpe(pos)
x = self.transformer.drop(tok_emb + pos_emb) #1
for block in self.transformer.h:
x = block(x) #2
x = self.transformer.ln_f(x) #3
logits = self.lm_head(x) #4
loss = None
if targets is not None:
loss=F.cross_entropy(logits.view(-1,logits.size(-1)),
targets.view(-1), ignore_index=-1)
return logits, loss
Browse the model definition to trace embeddings, repeated decoder blocks, final normalization, and the vocabulary projection. section 11.2.4
