A language modeling head projects a model’s final representations into vocabulary-sized logits. In DistilGPT2, it sits on GPT2LMHeadModel, above the base GPT2Model; after final layer normalization, it maps each vector to scores for 50,257 vocabulary tokens. section 3.1 section 3.1.1
The logits are scores, not a selected token. A probability distribution can be derived from them, and a next token can then be selected. For analysis, the same final normalization and head can project intermediate hidden states to show which tokens the model favors at each layer. section 3.1.1 section 10.3
Listing 10.10 The logit lens function
def logit_lens(model, tokenizer, hidden_states, topk=3):
final_norm = model.model.norm #A
lm_head = model.get_output_embeddings() #B
results = []
for hs in hidden_states[1:]: #C
last_token_hs = hs[0, -1, :] #1
normed = final_norm(last_token_hs)
logits = lm_head(normed)
probs = torch.softmax(logits.float(), dim=-1)
top = torch.topk(probs, topk)
tokens = [tokenizer.decode([idx]) for idx in top.indices.tolist()]
results.append(list(zip(tokens, top.values.tolist())))
return results
This function applies the model’s final normalization and output head to each layer’s last-token hidden state, then returns its top-k tokens and probabilities. section 10.3
Comparing layer predictions
For “The most significant building in Barcelona is the,” the logit lens records the top three predicted tokens and their probabilities at each layer. section 10.3

