A collate function merges individual tokenized samples into one training batch. Instruction fine-tuning needs a custom function because sequences require task-specific padding, targets, and loss masking. section 7.3
A custom collate function prepares the model-facing tensors in a fixed sequence:
- Find the longest sample, append the end-of-text padding token 50256, and pad every sequence to the batch length. section 7.3
Three variable-length samples
For [0,1,2,3,4], [5,6], and [7,8,9], the input batch becomes [[0,1,2,3,4], [5,6,50256,50256,50256], [7,8,9,50256,50256]]. Targets shift each row one position and replace extra padding targets with -100.
