During the 2000s, high-performing computer-vision systems commonly relied on handcrafted feature pipelines. Researchers designed image features, combined them into representations, then passed those representations to shallow learning algorithms. This approach remained competitive through 2012. section 2.1
The pipeline separates feature extraction from prediction: local descriptors describe image patches, while methods such as bag-of-visual-words and spatial pyramids build global image representations. A classifier maps those engineered features to outputs rather than learning intermediate representations. section 2.1
Hand-designed features encode particular assumptions about visual patterns:
- SIFT was designed to account for changes in scale and rotation. section 2.1
- HOG captures edge patterns useful for shape detection. section 2.1
These features were reliable and interpretable, but their assumptions were fixed in advance. A classifier could faithfully separate its inputs while the engineered features represented an irrelevant pattern as meaningful. section 2.1
Ripples mistaken for a car
An object detector labels a patch of rippled water beside a swimming goose as a car. The HOG features of that patch resemble car-associated features, illustrating how a failure can originate in the representation supplied to the classifier. section 2.1
Figure 2.1 This figure shows a goose swimming (top), where an object detector mistakenly labels a small patch of rippled water as a car. Below the image are three visualizations that clarify the source of this error: a close-up of the water patch (left), the corresponding HOG features (middle), and a specialized visualization (right) showing that the rippled water’s features resemble those associated with cars in HOG feature space. Used with permission granted by the lead author (Carl Vondrick).
Compare the water patch with its HOG features and the visualization of its similarity to car-associated features. section 2.1
Even in 2011, handcrafted pipelines remained the benchmark standard, and researchers demanded strong challenge results before accepting neural networks as a convincing alternative. section 2.2
