liveBook · Manning

Inner product

As explained in Understanding AI

A dot product is a scalar obtained by multiplying corresponding components of two vectors and summing the products.

A dot product turns two vectors into one scalar: multiply corresponding components, then add the results. In attention, that scalar measures how closely a query matches a key. section 2.1.3

Finding what “it” refers to

When processing “The performance was unbelievable—the crowd couldn't believe it,” the query for “it” is compared with every token's key. A strong dot-product score for “performance” signals high relevance; weaker scores for “crowd” and “couldn't” signal less relevance. Softmax then converts these scores into attention weights. section 2.1.3

Figure 2.3 When processing "it," the model assigns an attention weight to every other token in the sentence. "Performance" receives the highest weight by far (0.68 out of 1.0)which is how the model infers what "it" refers to.

See how attention distributes weights across the sentence, giving the strongest weight to “performance” when processing “it.” section 2.1.3

How Understanding AI covers it

How deep it goes
Defines componentwise multiplication and summation, then applies the result to attention without deriving vector arithmetic.
Examples
Uses token matching in a sentence to show how “it” relates most strongly to “performance.”
What you can do after
Understand how attention converts query-key similarity into relevance weights.
How it connects
Connects vectors, queries, keys, values, attention, and softmax.

Also explained in

Compare how 19 books explain inner product →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.