liveBook · Manning

Token shifting

As explained in Rearchitecting LLMs

Token shifting aligns each input position with the subsequent token that a causal language model must predict.

Causal language models predict the next token at each position. To calculate task loss, compare each logit position with the following label: drop the final logit, which has no next token, and the first label, which has no preceding prediction. Mark padding labels with -100 so cross-entropy ignores them. section 6.3.1

Aligning a sequence

For “Work hard play hard,” compare the logits for “Work,” “hard,” and “play” with the labels “hard,” “play,” and “hard”; the final logit has no target. section 6.3.1

shift_logits = student_logits[..., :-1, :].contiguous() #A
shift_labels = labels[..., 1:].contiguous() #B
loss_task = F.cross_entropy(
    shift_logits.view(-1, shift_logits.size(-1)), #C
    shift_labels.view(-1),
    ignore_index=-100 #D
)

The PyTorch listing slices logits and labels into next-token pairs before flattening them for cross-entropy. section 6.3.1

How Rearchitecting LLMs covers it

How deep it goes
Shows the logit and label slices and padding mask, without detailing alternative alignment schemes.
Examples
Aligns predictions and targets using the sequence “Work hard play hard.”
What it assumes
Assumes next-token prediction and explains how to align its logits and labels.
What you can do after
Apply the alignment in PyTorch by slicing logits and labels before cross-entropy.
How it connects
Connects causal language modeling with cross-entropy loss and knowledge distillation.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.