The Bitter Lesson is the argument that general-purpose methods which reliably turn more computation into better performance tend to outlast hand-tuned heuristics and domain-specific strategies. Encoding prior knowledge can help in the short term, but learned features can surpass handcrafted ones as data and computation grow. section 3.8
This is not simply a claim that more compute wins. Scaling pays off when a method can convert added resources into performance. Residual connections, for example, matter in deep learning because they support learning layered representations across many layers; adding compute to a convex optimization problem does not create the same scaling loop. section 3.8
Handcrafted features and learned representations
AlexNet’s success over handcrafted feature engineering such as SIFT illustrates the lesson: prior knowledge may offer early gains, while architectures that learn features from data can surpass them at scale. section 3.8
