An inner product multiplies corresponding entries in two equally sized vectors and adds the results, producing a scalar. In tensor libraries, the same operation appears as a dot product or matmul. section 2.3.3
From vectors to attention
For vectors, implement the operation with NumPy's np.matmul, Keras's keras.ops.matmul, or Python's @ operator. In attention, use the resulting dot product as a relevance score between target and source vectors, normalize scores with softmax, and compute a weighted sum of source vectors. section 2.3.3 section 15.3.1
Keep these shape and interpretation rules in mind:
- Require vectors to have the same number of elements; their product returns a scalar. section 2.3.3
- Treat a matrix-vector product as a vector of inner products between the matrix rows and the vector. section 2.3.3
- For matrix multiplication, require the first matrix's width to match the second matrix's height; the result combines row-column inner products. section 2.3.3
- Do not assume matmul is symmetric once either operand has more than one dimension. section 2.3.3
x = np.random.random((32,)) y = np.random.random((32,)) z = np.matmul(x, y) #1 z = x @ y #2
Run the compact NumPy and @ forms to see a vector inner product produce a scalar. section 2.3.3
def dot_product_attention(target, source): scores = np.einsum("btd,bsd->bts", target, source) #1 scores = softmax(scores, axis=-1) return np.einsum("bts,bsd->btd", scores, source) #2 dot_product_attention(target, source)
Inspect batched dot products in attention: einsum creates target-source scores, softmax normalizes them, and a second einsum forms the weighted output. section 15.3.1
