liveBook · Manning

Language modeling head

As explained in Rearchitecting LLMs

A language modeling head maps model representations to scores over a language model's vocabulary.

A language modeling head projects a model’s final representations into vocabulary-sized logits. In DistilGPT2, it sits on GPT2LMHeadModel, above the base GPT2Model; after final layer normalization, it maps each vector to scores for 50,257 vocabulary tokens. section 3.1 section 3.1.1

The logits are scores, not a selected token. A probability distribution can be derived from them, and a next token can then be selected. For analysis, the same final normalization and head can project intermediate hidden states to show which tokens the model favors at each layer. section 3.1.1 section 10.3

Listing 10.10 The logit lens function
def logit_lens(model, tokenizer, hidden_states, topk=3):
    final_norm = model.model.norm #A
    lm_head = model.get_output_embeddings() #B
    results = []
    for hs in hidden_states[1:]: #C
        last_token_hs = hs[0, -1, :] #1
        normed = final_norm(last_token_hs)
        logits = lm_head(normed)
        probs = torch.softmax(logits.float(), dim=-1)
        top = torch.topk(probs, topk)
        tokens = [tokenizer.decode([idx]) for idx in top.indices.tolist()]
        results.append(list(zip(tokens, top.values.tolist())))
    return results

This function applies the model’s final normalization and output head to each layer’s last-token hidden state, then returns its top-k tokens and probabilities. section 10.3

Comparing layer predictions

For “The most significant building in Barcelona is the,” the logit lens records the top three predicted tokens and their probabilities at each layer. section 10.3

How Rearchitecting LLMs covers it

How deep it goes
Explains vocabulary projection and demonstrates reuse on intermediate hidden states, without detailing the head’s internal implementation.
Examples
Uses DistilGPT2’s 50,257-token vocabulary and a Barcelona prompt to compare top predictions across layers.
What you can do after
Understand intermediate-layer token preferences by applying final normalization and the output head to hidden states.
How it connects
Connects the head to Transformer hidden states, layer normalization, logits, softmax, and the logit lens.

Also explained in

Compare how 2 books explain language modeling head →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.