Word order matters when a sentence says more than a few keywords can convey. A convolutional layer helps a neural network find short patterns of words wherever they occur in a sequence. Think of its filter, or kernel, as a stencil sliding across numerical representations of the words: each position produces a match score. Those scores form a feature map. section 7.1 section 7.2.1
A kernel has a width—the number of token positions it examines at once—and a stride—the number of positions it moves each step. With stride 1, a two-word kernel checks every adjacent pair. Padding short inputs and clipping long ones to a chosen length make their resulting representations compatible with later layers; clipping also means patterns beyond that limit cannot be found. section 7.2.2 section 7.4.1
Inspect a pattern before training one
Represent each word by four part-of-speech indicators: adverb, adjective, verb, and noun. A three-position kernel with hand-set weights can score the pattern adverb → verb → noun. With PyTorch, construct `torch.nn.Conv1d(in_channels=4, out_channels=1, kernel_size=3, bias=False)`, load those weights, and run the tagged sentence through it. The span “rightly timed pause” receives a score of 3. This experiment makes the layer’s inputs, weights, and position-by-position output visible before asking training to learn the weights. section 7.2.6
Listing 7.8 Loading hardcoded weights into a Conv1d layer
>>> kernel = torch.tensor(kernel.values, dtype=torch.float32)
>>> kernel = kernel.unsqueeze(0) #1
>>> conv = torch.nn.Conv1d(in_channels=4,
... out_channels=1,
... kernel_size=3,
... bias=False)
>>> conv.load_state_dict({'weight': kernel})
>>> print(conv.weight)
tensor([[[1., 0., 0.],
[0., 0., 0.],
[0., 1., 0.],
[0., 0., 1.]]])
Inspect how a four-channel, three-position pattern becomes the weight tensor of a PyTorch Conv1d layer. section 7.2.6
For text, the convolution should move along the token sequence, not across the components of each word embedding. Treat embedding dimensions as input channels and transpose embedding-layer output as needed to align them with PyTorch’s convolutional channels. Unlike the hand-set example, a trained layer adjusts its filter weights through backpropagation to find patterns useful for its labeled task. section 7.4.2 section 7.4.3
In a text classifier, keep track of what the layer produces and what comes next:
- Tokenize, pad or clip, and turn token IDs into embedding vectors; pretrained GloVe vectors or learned embeddings can supply those representations. section 7.4.1 section 7.4.2
- Use kernels of several widths to detect patterns across different numbers of words. Convolution produces a sequence of scores, not yet a single document prediction. section 7.5.1 section 7.2.4
- Apply pooling to reduce those scores, then feed the result to a prediction layer. Check tensor lengths: without padding, a stride-1 kernel of width K shortens a sequence of length N to N − K + 1. section 7.5.1 section 7.4.1
A disaster-tweet classifier is one practical use: its filters can learn local cues relevant to a label rather than relying only on which words appear. Evaluate it on held-out posts, not training accuracy alone. In one training run, test accuracy fell while training accuracy kept rising—a warning that more training did not improve generalization. section 7.5 section 7.5.1
