liveBook · Manning

Intermediate representation

As explained in Rearchitecting LLMs

An intermediate representation is a vector-valued activation produced inside a neural model between its input and output layers.

An intermediate representation is a vector-valued activation produced inside a neural model between its input and output layers. In distillation, hidden states provide layer-level signals for aligning a student with a teacher. section 6.3.1

A layer map pairs student and teacher layers, which may have different indices. The hidden-state loss flattens and normalizes corresponding activations, then uses cosine similarity to measure their alignment. section 6.3.1

if gamma > 0:
    loss_hidden = 0.0
    num_aligned_layers = len(layer_map)
 
    for student_idx, teacher_idx in enumerate(layer_map):
        student_h = student_hiddens[student_idx]
        teacher_h = teacher_hiddens[teacher_idx]
 
        student_flat = student_h.reshape(
           -1, 
           student_h.size(-1)) #A
        teacher_flat = teacher_h.reshape(
           -1, 
           teacher_h.size(-1)) #A
 
        student_norm = F.normalize(
           student_flat, 
           p=2, 
           dim=1) #B
        teacher_norm = F.normalize(
           teacher_flat, 
           p=2, 
           dim=1) #B
 
        cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
        loss_hidden += (1 - cos_sim) #D
 
    loss_hidden = loss_hidden / num_aligned_layers #E
else:
    loss_hidden = torch.tensor(0.0, device=student_logits.device)

The implementation accumulates cosine-distance losses across mapped student and teacher layers. section 6.3.1

Feature Dynamics Distillation compares changes between consecutive layers rather than only comparing absolute hidden states. It aligns the student and teacher deltas with cosine similarity, emphasizing the direction of change rather than its magnitude. section 6.4.2

for student_idx in range(len(layer_map) - 1): #A
    teacher_idx = layer_map[student_idx]
    teacher_idx_next = layer_map[student_idx + 1]
 
    student_delta = student_hiddens[student_idx + 1] - #B
➥student_hiddens[student_idx]
    teacher_delta = teacher_hiddens[teacher_idx_next] - 
➥teacher_hiddens[teacher_idx]
 
    student_delta_flat = 
➥student_delta.reshape(-1, student_delta.size(-1)) #C
    teacher_delta_flat = teacher_delta.reshape(-1, teacher_delta.size(-1))
 
    student_delta_norm = 
➥F.normalize(student_delta_flat, p=2, dim=1) #D
    teacher_delta_norm = F.normalize(teacher_delta_flat, p=2, dim=1)
 
    cos_sim_deriv = (student_delta_norm * 
➥teacher_delta_norm).sum(dim=1).mean() #E
    loss_derivative += (1 - cos_sim_deriv)

The code computes mapped layer-to-layer deltas, normalizes them, and accumulates their cosine-distance loss. section 6.4.2

How Rearchitecting LLMs covers it

How deep it goes
Explains mapped-layer cosine alignment and extends it to changes between consecutive layers.
Examples
Compares student layer 5 with teacher layer 9, then aligns changes across mapped consecutive layers.
What it assumes
Assumes teacher and student hidden-state tensors plus a layer map; does not explain how to obtain them.
What you can do after
Build it from scratch by computing mapped hidden-state cosine loss and consecutive-layer delta alignment.
How it connects
Links intermediate representations to knowledge distillation, hidden states, cosine similarity, and compound loss.

Also explained in

Compare how 3 books explain intermediate representation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.