liveBook · Manning

Loss masking

As explained in Rearchitecting LLMs

Label masking excludes selected token positions from contributing to a model's training loss while leaving the input available to the model.

Loss masking excludes selected token positions from the training objective without removing them from the model’s input. For tool-call distillation, the prompt remains in `input_ids`, while its label positions are set to `-100`; completion labels remain active. section 13.4.2

A prompt-completion example is prepared by:

  • Joining and tokenizing the prompt and completion, then tokenizing the prompt separately to find its length. section 13.4.2
  • Copying the input token IDs into labels and replacing prompt positions with `-100`, bounded by the truncated sequence length. section 13.4.2
  • Padding labels with `-100` when batching, so padding positions are excluded too. section 13.4.2
Listing 13.14 Building the distillation examples
def build_distillation_example(prompt, completion, tokenizer,  
        max_length=MAX_TRAIN_LENGTH):
 
    full_text = prompt + completion #1
    full_encoded = tokenizer(
        full_text, truncation=True, max_length=max_length
    )
    prompt_ids = tokenizer(
        prompt, truncation=True, max_length=max_length
    )["input_ids"]
 
    prompt_len = len(prompt_ids) #2
    labels = list(full_encoded["input_ids"]) 
 
    for i in range(min(prompt_len, len(labels))): 
        labels[i] = -100 #3
 
    return {
        "input_ids": full_encoded["input_ids"],
        "attention_mask": full_encoded["attention_mask"],
        "labels": labels,
    }
 
distillation_examples = [
    build_distillation_example(ex["prompt"], ex["completion"], 
        tokenizer)
    for ex in train_dataset
]
distillation_dataset = Dataset.from_list(distillation_examples) #4

See the function that tokenizes the joined text and constructs labels with prompt positions masked. section 13.4.2

How Rearchitecting LLMs covers it

How deep it goes
Builds labels from tokenized inputs, handles truncation, and masks prompt and batch-padding positions.
Examples
Uses prompt-completion pairs, masks prompt labels with `-100`, and also masks padding labels during batching.
What it assumes
Requires prompt-completion pairs and tokenization; constructs the labels and batches from those inputs.
What you can do after
Apply it by constructing labels that exclude prompt and padding positions from the training loss.
How it connects
Connects knowledge distillation with tokenization, supervised fine-tuning, and batch collation.

Also explained in

Compare how 2 books explain loss masking →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.