liveBook · Manning

Inner product

As explained in Machines that Think

A dot product is a multiplication and summation operation on two vectors that produces a scalar measuring their alignment.

A dot product multiplies two vectors and produces a scalar. In self-attention, stronger alignment between a token’s query and another token’s key produces a higher relevance score. section 4.1.3

These raw scores are not attention weights: the model applies softmax to each row of scores to create weights that sum to one. section 4.1.3

Comparing token representations

The query for “rises” may align more strongly with the key for “sun” than with the key for “the,” giving “sun” a higher relevance score. section 4.1.3

Figure 4.4 The query for “rises” is compared with the keys of other tokens to produce relevance scores. A stronger match with “sun” gives it a higher score than “the,” indicating that it is more useful for interpreting “rises.”

See how comparing the query for “rises” with other tokens’ keys produces different relevance scores. section 4.1.3

How Machines that Think covers it

How deep it goes
Defines the operation as vector multiplication yielding a scalar and explains how alignment affects relevance scores.
Examples
Compares the query for “rises” with the keys for “sun” and “the.”
What you can do after
Understand how query-key alignment contributes to token relevance scores before softmax converts them into attention weights.
How it connects
Links query-key comparisons to self-attention scores and softmax-normalized attention weights.

Also explained in

Compare how 19 books explain inner product →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.