liveBook · Manning

Loss masking

Also called label masking

Label masking is a training technique that selectively masks tokens, labels, or conditioning signals to control which information is available to or contributes to the learning objective.

How 2 Manning books explain it

Rearchitecting LLMs

Label masking excludes selected token positions from contributing to a model's training loss while leaving the input available to the model.

How it is used
Prepares distillation examples so the model learns from tool-call completions rather than prompt tokens.
How deep it goes
Builds labels from tokenized inputs, handles truncation, and masks prompt and batch-padding positions.
Examples
Uses prompt-completion pairs, masks prompt labels with `-100`, and also masks padding labels during batching.
What you can do after
Apply it by constructing labels that exclude prompt and padding positions from the training loss.

Read how Rearchitecting LLMs explains loss masking →

Build a Text-to-Image Generator (from Scratch)

Label masking is the probabilistic removal of class-label conditioning during training so a model learns both conditional and unconditional predictions.

How it is used
Trains conditional and unconditional diffusion predictions simultaneously for classifier-free guidance.
How deep it goes
Explains masking through Bernoulli sampling, one-hot transformation, embedding layers, tensor shapes, and a modification exercise.
Examples
Uses clothing labels [2, 9, 0], ten-class one-hot tensors, two embedding outputs, and an exercise with [1, 3, 5, 7].
What you can do after
Implement label masking and inspect conditional embedding tensors with PyTorch.

Read how Build a Text-to-Image Generator (from Scratch) explains loss masking →

Also covered in

What to know first

Related concepts

Questions about loss masking

What is loss masking?

Label masking is a training technique that selectively masks tokens, labels, or conditioning signals to control which information is available to or contributes to the learning objective.

Which Manning books explain loss masking?

Rearchitecting LLMs and Build a Text-to-Image Generator (from Scratch).

What should I know before learning loss masking?

Manning books that teach loss masking usually introduce objective function first.

What is loss masking part of?

Loss masking is part of supervised fine-tuning. Supervised fine-tuning is further training a pretrained model on labeled input-output examples to adapt its behavior to desired outputs.

What is loss masking also called?

Loss masking is also called label masking.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.