Collate function
A collate function combines individual data samples into a batch, optionally transforming or padding them into a common format.
Inside Deep Learning: Math, Algorithms, Models
A data-loading callback that assembles individual dataset items into a batch.
- How it is used
- Assembles Dataset items into DataLoader batches when default collation cannot handle variable shapes, lengths, or custom targets.
- How deep it goes
- Explains callback design through padding, packing, permutation, and list-preserving transformations, without detailing DataLoader internals.
- Examples
- Uses RNN padding and packing, English-French translation batches, MixUp interpolation, and variable-sized detection images.
- What you can do after
- Build it from scratch by writing a callback that transforms Dataset outputs into the exact batch structure a model and training loop require.
Read how Inside Deep Learning: Math, Algorithms, Models explains collate function →
Build a Large Language Model (From Scratch)
A data-loading function that merges a list of individual samples into a single batch, optionally applying padding and other formatting.
- How it is used
- Prepares instruction-response samples for efficient transformer training through PyTorch DataLoader batches.
- How deep it goes
- Details padding, shifted targets, device placement, maximum length, and loss masking, but not DataLoader internals.
- Examples
- Demonstrates variable-length token lists, padded input tensors, shifted targets, and -100 loss masking.
- What you can do after
- Build it from scratch as a PyTorch collate function returning padded input and target tensors for instruction fine-tuning.
Read how Build a Large Language Model (From Scratch) explains collate function →
Related concepts
- DataLoaderuses · PyTorch DataLoader is an iterable utility that retrieves dataset samples, optionally shuffles and transforms them, and groups them into batches.
- batch sizeproduces · Batch size is the number of items grouped together for processing or computation as a single batch.
- MixUpused by · A data augmentation and regularization technique that trains on convex combinations of input examples and their labels.
- maskuses · Masking is the selective representation, concealment, inclusion, exclusion, or restriction of parts of data or a system according to specified criteria.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.