An inner product, commonly called a dot product, multiplies corresponding components of two vectors and adds the results. It turns a pair of vectors into one score: positive values suggest alignment, zero suggests no alignment, and negative values suggest opposition. section 9.20 section 11.19
The same operation appears in several practical settings:
- A neuron uses the dot product of its inputs and weights to replace a longer weighted-sum loop, before adding a bias. section 9.20
- Self-attention dot-products each Query with every Key to produce relevance scores between tokens. section 11.19
- A convolutional filter dot-products its weights with each image patch while scanning across the image. section 12.14
A small calculation
For filter row [2, 7, 2] and pixel row [11, 2, 1], multiply matching positions and add: (2 × 11) + (7 × 2) + (2 × 1) = 38. A CNN repeats this calculation as the filter moves across the image. section 12.14
Figure 11.36 An analogy of how self-attention works
Use the alignment analogy to see why a Query-Key dot product can represent relevance: similarly pointed vectors score highly, while opposed vectors score negatively. section 11.19
