Text sequences rarely have the same length. A padding token fills unused positions in shorter sequences so they can be represented at a chosen fixed length. CNNs use padding alongside clipping to produce consistently sized inputs; transformers use it to keep input shapes consistent across stacked layers. Padding does not add words to the original text. section 7.4.1 section 9.1.2
Choose a marker that will not be mistaken for a word in your dataset, such as `<PAD>` or `<pad>`, and use it consistently in the vocabulary and embedding layer. ID 0 is a common choice, not a requirement. For a CNN, padding can go at the beginning or end, or be split between them; do not insert it between words, where it would interfere with convolution. Clip sequences that exceed the chosen length. section 7.4.1 section 9.2.3
Pad a short sequence
A three-token sequence padded to length five needs two filler positions. One practical function first clips to `seq_len`, then appends `[pad_value] * (seq_len - len(padded))`. With `pad_value='<PAD>'`, `pad(['forest', 'fire', 'near'], '<PAD>', 5)` yields `['forest', 'fire', 'near', '<PAD>', '<PAD>']`. section 7.4.1
In translation, source and target tokenizers can each register `<pad>` as a special token. The model must receive the corresponding padding ID for each vocabulary. That ID also identifies filler positions when constructing source and target key-padding masks; the source mask is reused for encoder memory. Training loss can ignore target padding IDs so filler positions do not count as targets. section 9.2.3
A padding token is not a mask token: `<mask>` intentionally hides a known token for an encoder training task, whereas `<pad>` fills positions beyond a shorter example. Nor does padding eliminate the cost of processing a fixed-length input. Choose a maximum length with both coverage and the extra computation and memory for filler positions in mind. section 9.2.3 section 9.1.2
Listing 7.18 A multipurpose padding function
def pad(sequence, pad_value, seq_len):
padded = list(sequence)[:seq_len]
padded = padded + [pad_value] * (seq_len - len(padded))
return padded
The short Python function shows clipping and right-padding in the same operation. section 7.4.1
Listing 9.9 TranslationTransformer.prepare_src()
>>> def _make_key_padding_mask(self, t, pad_idx): ... mask = (t == pad_idx).to(self.device) ... return mask ... ... def prepare_src(self, src, src_pad_idx): ... src_key_padding_mask = self._make_key_padding_mask( ... src, src_pad_idx) ... src = rearrange(src, 'N S -> S N') ... src = self.pos_enc(self.src_emb(src) ... * math.sqrt(self.d_model)) ... return src, src_key_padding_mask
The source-preparation code shows how a padding ID becomes a transformer key-padding mask. section 9.2.3
