liveBook · Manning

Scaled-dot product attention

As explained in Build AI Drug Discovery Pipelines

Scaled dot-product attention computes attention weights from query-key dot products divided by the square root of the key dimension and applies them to value vectors.

Scaled-dot product attention compares each query with every key using a dot product, divides scores by √dₖ, normalizes them with softmax, and combines the value vectors using the resulting weights. section 12.3.2

Attention(Q,K,V)=softmax(QKTdk)V

The compact equation shows the complete operation: similarity scores become normalized attention weights, which produce a weighted value representation. section 12.3.2

A distant cysteine partner

For residue 1, q₁ = [1, 0] gives raw scores 1.0, 0.0, and 0.9 against three keys. Scaling by √2 and applying softmax produces weights of about 0.41, 0.20, and 0.38, so the new representation strongly incorporates residue 3 despite its sequence distance. section 12.3.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains scoring, scaling, softmax normalization, value aggregation, and final linear projection, but stops before implementation code.
Examples
Demonstrates three two-dimensional residue vectors and a distant cysteine pair with explicit scores, weights, and weighted values.
What it assumes
Assumes familiarity with query, key, and value vectors, while defining their attention roles through a concrete residue example.
What you can do after
Evaluate/understand how attention weights stabilize training and combine context from sequence-distant residues.
How it connects
Connects queries, keys, values, softmax, self-attention, cross-attention, causal self-attention, and linear projection.

Also explained in

Compare how 6 books explain scaled-dot product attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.