AlexNet’s 2012 ImageNet result made error rate a turning point in computer vision. Its top-5 error rate was 15.3%, against 26.2% for the runner-up. This did not mean every first guess was right: top-5 error counts an image as an error only when its correct label is absent from the model’s five highest-ranked predictions. section 2.0 section 2.5
A correct fourth guess
Suppose an image shows a basketball. A model guesses “croquet ball,” “bikini,” “warthog,” “basketball,” and “moving van,” in that order. Its first guess is wrong, but this is not a top-5 error: “basketball” appears fourth. The distinction matters whenever error rates are compared. section 3.6
The ImageNet trend gave the metric historical force. Handcrafted-feature approaches had plateaued; AlexNet’s convolutional neural network sharply reduced top-5 error in 2012. GoogLeNet reached 6.7% in 2014, and a ResNet ensemble reached 3.57% in 2015. These numbers tracked rapid progress on the competition’s classification task, not every kind of visual understanding. section 2.5.5 section 3.0
Figure 2.6 The top-5 error rates in the ILSVRC from 2010 to 2017. SIFT and SVM won in 2010 and 2011, but performance plateaued. CNNs won from 2012 to 2017, including AlexNet in 2012, and eventually saturated the benchmark.
The 2010–2017 ILSVRC chart makes the plateau before AlexNet and the subsequent decline in top-5 error visible. section 2.5.5
A low benchmark error rate needs a carefully stated comparison. ImageNet’s top-5 rule gives a machine credit for including the right label among several guesses; people do not ordinarily answer that way. A widely repeated 5% human-performance threshold came from just two annotators, trained on 500 and 100 images respectively. ResNet’s success was substantial, but calling it “human-level” on that basis was methodologically questionable. section 3.6
Training and test error answer a different question: whether performance on training examples carries over to held-out data. Growing model capacity can drive training error down without reliably improving test performance. In overparameterized settings, test error can also fall again beyond the point where a model fits every training example, complicating the familiar U-shaped account of overfitting. section 8.5.2
