Masking prevents padded positions from affecting training. In batched GPT inputs, shorter sequences receive padding tokens so every sequence has the same length; attention masking keeps those positions from being attended to. section 2.4
Loss masking is a related operation applied to target token IDs. Replacing padding targets with -100 makes PyTorch cross-entropy ignore them through its default ignore_index=-100 setting. The first end-of-text target remains active so the model can learn when a response ends. section 7.3
A batching pipeline applies masking after creating aligned inputs and targets:
- Pad each sequence with token ID 50256, then form inputs by removing the final position and targets by shifting one position. section 7.3
- Find target positions containing padding, preserve the first end-of-text target, and replace later padding targets with -100. section 7.3
- Optionally replace instruction and input targets with -100 so loss trains the generated response rather than the prompt. section 7.3
Masked target batch
For inputs [[0, 1, 2, 3, 4], [5, 6, 50256, 50256, 50256]], aligned targets become [[1, 2, 3, 4, 50256], [6, 50256, -100, -100, -100]]. Cross-entropy ignores the -100 entries, while still learning the meaningful end-of-text target. section 7.3
Figure 7.5 The three-stage process for instruction fine-tuning an LLM. Next, we look at step 2 of stage 1: assembling the training batches.
Use the instruction fine-tuning pipeline figure to locate masking within batch assembly and the surrounding training stages. section 7.3
