Padding adds boundary values or placeholder positions so an operation can handle edges and variable-length data. In convolutional layers, zero padding adds rows of zeros around an image; in sequence batches, padding appends zeros to shorter token sequences. section 4.2.1 section 10.1.2
For images, padding changes how a filter reaches the border and helps control feature-map dimensions. With a 2 × 2 kernel, stride 2, and padding=1, a 3 × 3 input becomes a 5 × 5 padded image before convolution. For text, padding makes every sequence in a batch equally long, while masks prevent padded positions from affecting the Transformer. section 4.2.2 section 10.1.2
A practical padding workflow includes:
- Choose the padding amount for a convolution; padding=0 leaves the input unchanged, while padding=1 adds one zero row around every side. section 4.2.1 section 4.2.2
- Pad each sequence to the longest sequence in its batch, appending zeros to shorter numerical representations. section 10.1.2
- Sort sequences by length before batching to reduce the amount of padding and accelerate training. section 10.1.2
A padded convolution
Using the image [[1,1,1],[0,1,2],[8,7,6]], weights [[1,2],[3,4]], stride=2, and padding=1 produces the output [[4,7],[32,50]]. The top-left value uses the padded region [[0,0],[0,1]], so only the input value 1 contributes to that corner calculation. section 4.2.2
