Feature engineering means constructing or transforming inputs so a model can use them. In an NLP pipeline, that might mean turning text into word-based vectors, part-of-speech tags, or patterns detected across a sequence. The features shape what the next modeling stage can recognize. section 1.9 section 5.1.2
Hand-designed features are useful when you know what to look for. TF-IDF vectors, for example, work well for information retrieval and full-text search, but may be less useful for semantic search when words are misspelled or ambiguous. A representation that serves one task need not serve another. section 5.1.2
When a model misses a pattern, possible transformations include:
- Applying a nonlinear function, such as a square, cube, square root, logarithm, or exponential, then checking whether prediction errors improve. section 5.1.2
- Combining inputs to capture interactions. Polynomial features can do this, but the number of candidates grows rapidly as more interactions are considered. section 5.1.2
- Encoding a known sequence pattern in a convolution kernel. This works best when you can specify the pattern you want to detect. section 7.2.5
An adverb pattern as a feature
Tag a sentence for parts of speech, then encode each token as 1 if it is an adverb and 0 otherwise. Sliding the kernel [.5, .5] across that sequence produces a two-token average: adjacent adverbs produce 1, while a window containing one adverb produces .5. The output is a new feature sequence describing local adverb patterns. section 7.2.5
Listing 7.4 Convolution in pure Python
>>> kernel = [.5, .5] #1
>>>
>>> output = []
>>> for i in range(len(inpt) - 1): #2
... z = 0
... for k, weight in enumerate(kernel): #3
... z = z + weight * inpt[i + k]
... output.append(z)
>>>
>>> print(f'inpt:\n{inpt}')
>>> print(f'len(inpt): {len(inpt)}')
>>> print(f'output:\n{[int(o) if int(o)==o else o for o in output]}')
>>> print(f'len(output): {len(output)}')inpt:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0., 1, 1., 0, 0, 0., 1., 0, 0, 0]
len(inpt): 20
output:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, .5, 1, .5, 0, 0, .5, .5, 0, 0]
len(output): 19
The pure-Python convolution shows how a fixed kernel turns a token-level feature sequence into local averages. section 7.2.5
Neural networks offer another route: they learn feature extractors rather than requiring every transformation to be chosen by hand. That can search a broader range of representations, but learned extractors may need more training data than intentionally designed algorithms. A handcoded convolution makes the pattern explicit; a CNN can instead learn which patterns to seek. section 1.10 section 7.2.5
