2 The AlexNet Moment

 

This chapter covers

  • Skepticism about artificial neural networks
  • Feature engineering before AlexNet
  • Training artificial neural networks is hard
  • ImageNet’s role in AlexNet’s success
  • The AlexNet Moment and technical innovations
Papers
  • ImageNet Classification with Deep Convolutional Neural Networks (Krizhevsky, Sutskever, & Hinton, 2012)

In 2012, Ilya, alongside Alex Krizhevsky and Geoffrey Hinton, trained a convolutional neural network to classify images. Their network stunned the AI community by dramatically reducing error rates in the emerging ImageNet Large Scale Visual Recognition Challenge (ILSVRC). AlexNet achieved a top-5 error rate of 15%, significantly outperforming conventional methods reliant on handcrafted feature engineering, which produced around 26% error. Top-5 error rate measures how often the correct answer is not among the model’s top five guesses.

2.1 Feature Engineering vs. Representation Learning

2.2 Pre-AlexNet Skepticism

2.3 ImageNet

2.4 Why Training is Hard

2.5 AlexNet

2.5.1 Network Architecture

2.5.2 Training Innovations

2.5.3 Data Augmentation

2.5.4 Efficient and Scalable Training

2.5.5 The Fallout