An attention matrix is a probability matrix showing how strongly each token attends to every token in a sequence. In a vision transformer, each head produces one matrix for an image. section 3.4.3
A ViT attention matrix has these implementation-level properties:
- A batch of 32 images produces tensors shaped [32, 4, 65, 65] for each encoder layer. section 3.4.3
- Each 65 × 65 matrix represents one image’s CLS token and 64 image-patch tokens. section 3.4.3
- Each column sums to 1, representing a complete attention distribution. section 3.4.3
For visualization, attention maps from four layers are concatenated, CLS-to-patch values are selected, and values are averaged across heads and layers. The resulting 64 values become an 8 × 8 grid and are upscaled to 32 × 32.
