liveBook · Manning

Inner product

As explained in Timeless Algorithms, The Foundational Ideas

A dot product is an operation that maps two equal-length numerical vectors to the sum of their pairwise coordinate products.

An inner product, or dot product, combines two equal-length numerical vectors into one number. Multiply corresponding components, then add the products. The result can express a weighted feature calculation or a measure of alignment between vectors. section 8.4 section 12.2.1

wTx=w1x1+w2x2+…+wpxp

Here w and x each have p components. The transpose symbol T changes the orientation of w for multiplication, not its component values. section 8.4

In a fitted support vector machine (SVM), the dot product wᵀx combines feature values with their fitted weights; adding the bias b gives the decision score. Its sign determines the predicted class. Dot products also connect the direct and support-vector prediction forms; kernels calculate transformed dot products without explicitly constructing every new feature. section 8.4 section 8.9

A weighted classification score

Let w = (6, −1), b = −5, and x = (3, 9), where the features are temperature and temperature squared on a simplified scale. The dot product is (6 × 3) + (−1 × 9) = 9. Adding the bias gives a score of 4, so the prediction is class +1. The boundary is linear in this feature space, though the rule is nonlinear in temperature alone. section 8.4

Alignment in attention

Suppose the query for “it” is [0.2, 0.8], and the keys for “trophy” and “box” are [1, 0] and [0, 1]. Their dot products with the query are 0.2 and 0.8, respectively, indicating stronger compatibility with “box.” The example assumes identity projections. Dividing by √2 gives scaled scores of approximately 0.14 and 0.57; softmax then turns them into attention weights of approximately 0.39 and 0.61. The scores before softmax are not probabilities. section 12.2.1

These simplified attention vectors make the arithmetic visible, but a single calculation does not explain the behavior of a complete transformer, which also uses multiple attention heads, feed-forward networks, and stacked layers. section 12.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Works through component-wise arithmetic in both settings, while simplifying attention vectors rather than reproducing a complete transformer.
Examples
Calculates a temperature-based classification and compares pronoun-query compatibility with trophy and box keys.
What you can do after
Evaluate and understand SVM decision scores and attention compatibility through dot-product calculations.
How it connects
Connects dot products to support vector machines, kernels, self-attention, and softmax.

Also explained in

Compare how 19 books explain inner product →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.