liveBook · Manning

Hidden state

As explained in Rearchitecting LLMs

A hidden state is an internal vector representation produced by a neural model layer for the information processed up to that layer.

A hidden state is an internal vector representation produced as a model processes information. In a transformer, each block produces a representation for every token; those representations carry context from the sequence. section 9.6 section 10.3

Hidden states support more than final-token prediction. A mixture-of-experts router uses a token’s hidden state to choose experts, so the same token can be routed differently in different contexts. In knowledge distillation, a student can align its intermediate representations with a teacher’s alongside learning from hard labels and teacher logits. section 9.6 section 6.3

Request hidden states with output_hidden_states=True. The returned tuple begins with the embedding output, followed by one output per transformer block. Discard the first entry when comparing block outputs. A student with fewer blocks than its teacher also needs a layer mapping before their representations can be aligned. section 6.3.2 section 6.1.3

Inspecting predictions across layers

For Llama-3.2-1B, a forward pass produces 17 hidden-state entries: one embedding output and 16 block outputs. In the example prompt, each has shape (1, 9, 2048). A logit lens selects the last token’s vector at each block, applies the model’s final normalization and prediction head, then uses softmax and top-k to inspect favored vocabulary tokens. section 10.3

Listing 10.10 The logit lens function
def logit_lens(model, tokenizer, hidden_states, topk=3):
    final_norm = model.model.norm #A
    lm_head = model.get_output_embeddings() #B
    results = []
    for hs in hidden_states[1:]: #C
        last_token_hs = hs[0, -1, :] #1
        normed = final_norm(last_token_hs)
        logits = lm_head(normed)
        probs = torch.softmax(logits.float(), dim=-1)
        top = torch.topk(probs, topk)
        tokens = [tokenizer.decode([idx]) for idx in top.indices.tolist()]
        results.append(list(zip(tokens, top.values.tolist())))
    return results

The logit-lens function shows the per-layer indexing, projection, and top-k extraction steps. section 10.3

During distillation, run the teacher without gradients because its outputs are needed for comparison, not parameter updates. Request hidden states only when the hidden-state loss is enabled; gradient accumulation can reduce the memory needed for an effective larger batch. section 6.3.2

How Rearchitecting LLMs covers it

How deep it goes
Explains hidden-state extraction, tensor shapes, layer alignment requirements, and logit-lens projection.
Examples
Uses a 14-block student and 18-block teacher for distillation, and a Barcelona prompt to inspect layer predictions.
What you can do after
Diagnose predictions with a logit lens and apply hidden-state alignment during knowledge distillation.
How it connects
Connects hidden states to knowledge distillation, mixture-of-experts routing, and the logit lens.

Also explained in

Compare how 4 books explain hidden state →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.