max_length sets the fixed token-ID length for each sampled input sequence. In text generation loaders, it defines the size of each input chunk and its shifted target chunk; in classification loaders, it can cap or equalize sequence length for batching. section 2.6 section 6.3
For sliding-window language modeling, the dataset slices tokenized text into rows of exactly max_length token IDs, then forms targets by shifting those rows by one position. For classification, the dataset pads shorter sequences or truncates longer ones to a chosen maximum so batches stay rectangular. section 2.6 section 6.3
Used here to control these data-shaping steps:
- Creates fixed-size input chunks for each dataset row. section 2.6
- Defines the target window by shifting inputs one token ahead. section 2.6
- Lets batches stack cleanly in PyTorch DataLoader. section 2.6 section 6.3
- Caps or matches sequence length in classification data preparation. section 6.3
Sliding-window batch example
With max_length set to 4, one batch contains input IDs [40, 367, 2885, 1464] and targets [367, 2885, 1464, 1807]. The next batch shifts forward by one token when stride is 1. section 2.6
Padding and truncation example
A classification dataset can set max_length from the longest training sequence, then truncate validation and test examples to that length and pad shorter ones with a pad token. section 6.3
