liveBook · Manning

Cosine distance

As explained in Rearchitecting LLMs

Cosine distance measures dissimilarity between vectors based on their directional alignment.

Cosine distance measures directional difference between vectors: vectors [1, 2, 3] and [10, 20, 30] have the same direction despite different magnitudes. For hidden-state alignment, normalize student and teacher representation vectors, calculate one minus their cosine similarity, and average across mapped layers. section 6.3.1

if gamma > 0:
    loss_hidden = 0.0
    num_aligned_layers = len(layer_map)
 
    for student_idx, teacher_idx in enumerate(layer_map):
        student_h = student_hiddens[student_idx]
        teacher_h = teacher_hiddens[teacher_idx]
 
        student_flat = student_h.reshape(
           -1, 
           student_h.size(-1)) #A
        teacher_flat = teacher_h.reshape(
           -1, 
           teacher_h.size(-1)) #A
 
        student_norm = F.normalize(
           student_flat, 
           p=2, 
           dim=1) #B
        teacher_norm = F.normalize(
           teacher_flat, 
           p=2, 
           dim=1) #B
 
        cos_sim = (student_norm * teacher_norm).sum(dim=1).mean() #C
        loss_hidden += (1 - cos_sim) #D
 
    loss_hidden = loss_hidden / num_aligned_layers #E
else:
    loss_hidden = torch.tensor(0.0, device=student_logits.device)

The implementation flattens batch and sequence dimensions, normalizes each representation, and averages cosine distance across aligned layers. section 6.3.1

How Rearchitecting LLMs covers it

How deep it goes
Computes one minus mean cosine similarity after flattening and normalizing representations, then averages across mapped layers.
Examples
Compares [1, 2, 3] with [10, 20, 30] to illustrate equal direction despite different magnitudes.
What it assumes
Assumes student and teacher hidden representations and a mapping between their layers.
What you can do after
Apply it via PyTorch tensor operations to calculate hidden-state alignment loss across mapped layers.
How it connects
Connects cosine similarity to knowledge distillation and hidden-state alignment.

Also explained in

Compare how 9 books explain cosine distance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.