Tokenization converts raw text into numerical token IDs a language model can process. In training, it produces tensors such as input_ids, attention_mask, and labels. section 2.5
A training tokenization stage can:
- Pad or truncate text to a chosen maximum length, such as 128 tokens. section 2.5
- Copy input IDs into labels for the training dataset. section 2.5
- Check that paired prompts have equal token counts so corresponding positions can be compared. section 11.1.1
def tokenize_for_kd(examples, max_length=128):
texts = [ex["text"] for ex in examples]
tokenized = tokenizer(
texts,
padding="max_length",
truncation=True,
max_length=max_length,
return_tensors="pt"
)
return {
"input_ids": tokenized["input_ids"],
"attention_mask": tokenized["attention_mask"],
"labels": tokenized["input_ids"].clone()
}
# Process the raw text samples
recovery_samples = list(distillation_dataset) #1
tokenized_data = tokenize_for_kd(recovery_samples)
Example function that tokenizes text with padding and truncation, then returns input IDs, attention masks, and labels. section 2.5



































