A dot score ranks each state vector by its alignment with a context vector. Compute the dot product and divide by √H, where H is the vector dimension. Positive values indicate similarity, negative values indicate difference, and orthogonal vectors score zero. Scaling limits large logits that could produce vanishing gradients through softmax. section 10.2.1 section 10.2
For batched attention, map the mathematics directly to tensor shapes:
- Represent states as (B, T, H) and context as (B, H). section 10.2.1
- Unsqueeze context to (B, H, 1), then multiply (B, T, H) by (B, H, 1). section 10.2.1
- Return (B, T, 1): one score for each of T items, then use the scores in attention weighting. section 10.2.1
Reusable PyTorch module
class DotScore(nn.Module): def __init__(self, H): super().__init__() self.H = H def forward(self, states, context): return torch.bmm(states, context.unsqueeze(2)) / np.sqrt(self.H) This computes all T scores for every batch item in one operation. section 10.2.1
class DotScore(nn.Module):
def __init__(self, H):
"""
H: the number of dimensions coming into the dot score.
"""
super(DotScore, self).__init__()
self.H = H
def forward(self, states, context):
"""
states: (B, T, H) shape
context: (B, H) shape
output: (B, T, 1), giving a score to each of the T items based
on the context
"""
T = states.size(1) scores =
torch.bmm(states,context.unsqueeze(2))
/ np.sqrt(self.H) #1
return scores
#1 Computes <span class="math"><img alt="" src="../Images/eq_hh_t-top_bar_h.png" /></span>. (B, T, H) -> (B, T, 1).
Use the listing to inspect the complete tensor-shape contract and reusable implementation. section 10.2.1
