liveBook · Manning

Hard label

As explained in Rearchitecting LLMs

A hard label is a target representation that identifies the designated correct class or token without encoding alternative probabilities.

A hard label identifies the dataset-designated correct class or token. Unlike a soft label, it gives no probability distribution over alternatives. section 6.3 section 6.3.1

In knowledge distillation, hard labels provide the task objective: they encourage the student to predict the dataset’s correct answer. Combined with soft-label logits loss, they balance task accuracy against learning the teacher’s confidence across alternatives. section 6.3 section 6.5.1

Next-token targets

For “Work hard play hard,” the model’s prediction after “Work” is compared with “hard,” and each later prediction is aligned with the following token. The first label and final logit are omitted; padding labels marked -100 are excluded from cross-entropy. section 6.3.1

shift_logits = student_logits[..., :-1, :].contiguous() #A
shift_labels = labels[..., 1:].contiguous() #B
loss_task = F.cross_entropy(
    shift_logits.view(-1, shift_logits.size(-1)), #C
    shift_labels.view(-1),
    ignore_index=-100 #D
)

Inspect the PyTorch snippet for shifting logits and labels before cross-entropy, with padded labels ignored. section 6.3.1

How Rearchitecting LLMs covers it

How deep it goes
Implements dataset-target cross-entropy with next-token shifting and padding exclusion.
Examples
Uses “Work hard play hard” to illustrate next-token alignment and excludes padded targets marked -100.
What you can do after
Apply via PyTorch by shifting next-token logits and labels, then computing cross-entropy while excluding padded labels.
How it connects
Combines task loss with soft-label logits loss and hidden-state alignment in knowledge distillation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.