Dot-product attention scores each source vector against the current target vector, normalizes those scores with softmax, and combines source vectors into a relevance-weighted representation. In translation, it lets each predicted token focus on the most useful source words instead of carrying all information through a long recurrent loop. section 15.3.1
A complete attention step
For target and source sequences, compute every target–source dot product, normalize across source positions, then form weighted source sums: scores <- op_einsum("btd,bsd->bts", target, source) scores <- op_softmax(scores, axis = -1) combined <- op_einsum("bts,bsd->btd", scores, source) The score matrix has one row per target position; each row distributes one target token’s attention across the source sequence.
