Attention lets a model learn which inputs matter most, suppress irrelevant features, and combine the remaining information into a useful representation. It is especially useful when inputs contain multiple items, such as digits in a bag, words in a sentence, or image regions. section 10.0
A basic attention mechanism follows three learned-computation steps:
- Transform each input xt into a representation ht = F(xt), using a shared feature-extraction network. section 10.1.2 section 10.1.3
- Assign every representation a raw score, then apply softmax across the T items to obtain normalized weights αt. section 10.1.2
- Multiply each representation by its weight and sum the results: x̄ = Σt αt ht. section 10.1.2 section 10.1.3
Independent scoring makes each item’s importance depend only on itself. Context-sensitive attention instead scores an item using both its representation and a summary of the other inputs, commonly their average. This lets global information guide local weighting. section 10.2
Variable-length inputs require padding and a Boolean mask. Invalid items receive a very negative score before softmax, making their weights and contributions effectively zero. Valid-item averages must divide by the number of valid items, not the padded length. section 10.2.4 section 10.3
Largest-digit bags
A modified MNIST dataset groups T digit images into a bag and labels the bag with its largest digit. A backbone extracts one representation per image, attention learns which image is most relevant, and the weighted result feeds a classifier. Variable-size bags test whether masking preserves this behavior when padding is present. section 10.1 section 10.3
Sequence-to-sequence translation
An encoder produces hidden states for every source token. At each decoder step, the decoder state becomes the context for scoring those encoder states; the weighted context combines with the decoder state to predict the next target token. Saved weights can be plotted as an attention heat map. section 11.4 section 11.4.2
class Combiner(nn.Module):
"""
This class is used to combine a feature extraction network F and an
importance prediction network W, and combine their outputs by adding
and summing them together.
"""
def __init__(self, featureExtraction, weightSelection):
"""
featureExtraction: a network that takes an input of shape (B, T, D)
and outputs a new
representation of shape (B, T, D’).
weightSelection: a network that takes in an input of shape
(B, T, D’) and outputs
a tensor of shape (B, T, 1) or (B, T). It should be normalized,
so that the T
values at the end sum to one (torch.sum(_, dim=1) = 1.0)
"""
super(Combiner, self).__init__()
self.featureExtraction = featureExtraction
self.weightSelection = weightSelection
def forward(self, input):
"""
input: a tensor of shape (B, T, D)
return: a new tensor of shape (B, D’)
"""
features = self.featureExtraction(input) #1
weights = self.weightSelection(features) #2
if len(weights.shape) == 2: #3
weights.unsqueeze(2) #4
r = features*weights #5
return torch.sum(r, dim=1) #6
#1 (B, T, D) <span class="math"><i><b>h</b></i><i><sub>i</sub></i> = <i>F</i>(<i><b>x</b></i><i><sub>i</sub></i>)</span>
#2 (B, T) or (B, T, 1) for <span class="math"><i><b>α</b></i></span>
#3 (B, T) shape
#4 Now (B, T, 1) shape
#5 (B, T, D); computes <span class="math"><i>α</i><i><sub>i</sub></i> ⋅ <i><b>h</b></i><i><sub>i</sub></i></span>
#6 Sums over the T dimension, giving the (B, D) final shape
Use the Combiner implementation to connect a feature-extraction network with a normalized weight-selection network and inspect the required tensor shapes. section 10.1.3
