A padding token is a special marker appended to shorter sequences so every sequence in a batch has the same length. Unlike start, end, or unknown tokens, it represents no input content; it provides a fixed-size shape for batching. section 2.2.1 section 4.2.2
A typical padding workflow proceeds as follows:
- Append padding markers to shorter token sequences until they match the batch’s maximum or configured length. section 8.2.2
- Prevent computation from treating padding as content by supplying an attention mask with 0 at padded positions and 1 elsewhere. section 8.2.2
- Preserve the tokenizer’s expected special-token IDs when using pretrained weights, because those IDs are consumed internally by the model. section 12.2.1
Padding two captions
Tokenizing “two dogs run” and “an eagle flies in the sky” produces sequences of lengths five and eight after CLS and SEP are added. The shorter sequence receives three PAD tokens: [CLS, two, dogs, run, SEP, PAD, PAD, PAD]. Its attention mask is [1, 1, 1, 1, 1, 0, 0, 0], while the longer sequence needs no padding. section 8.2.2
