A hidden state is an internal vector representation produced as a model processes information. In a transformer, each block produces a representation for every token; those representations carry context from the sequence. section 9.6 section 10.3
Hidden states support more than final-token prediction. A mixture-of-experts router uses a token’s hidden state to choose experts, so the same token can be routed differently in different contexts. In knowledge distillation, a student can align its intermediate representations with a teacher’s alongside learning from hard labels and teacher logits. section 9.6 section 6.3
Request hidden states with output_hidden_states=True. The returned tuple begins with the embedding output, followed by one output per transformer block. Discard the first entry when comparing block outputs. A student with fewer blocks than its teacher also needs a layer mapping before their representations can be aligned. section 6.3.2 section 6.1.3
Inspecting predictions across layers
For Llama-3.2-1B, a forward pass produces 17 hidden-state entries: one embedding output and 16 block outputs. In the example prompt, each has shape (1, 9, 2048). A logit lens selects the last token’s vector at each block, applies the model’s final normalization and prediction head, then uses softmax and top-k to inspect favored vocabulary tokens. section 10.3
Listing 10.10 The logit lens function
def logit_lens(model, tokenizer, hidden_states, topk=3):
final_norm = model.model.norm #A
lm_head = model.get_output_embeddings() #B
results = []
for hs in hidden_states[1:]: #C
last_token_hs = hs[0, -1, :] #1
normed = final_norm(last_token_hs)
logits = lm_head(normed)
probs = torch.softmax(logits.float(), dim=-1)
top = torch.topk(probs, topk)
tokens = [tokenizer.decode([idx]) for idx in top.indices.tolist()]
results.append(list(zip(tokens, top.values.tolist())))
return results
The logit-lens function shows the per-layer indexing, projection, and top-k extraction steps. section 10.3
During distillation, run the teacher without gradients because its outputs are needed for comparison, not parameter updates. Request hidden states only when the hidden-state loss is enabled; gradient accumulation can reduce the memory needed for an effective larger batch. section 6.3.2



