chapter two

2 The AlexNet moment

 

This chapter covers

  • Skepticism about artificial neural networks
  • Feature engineering before AlexNet
  • The difficulty of training artificial neural networks
  • ImageNet’s role in AlexNet’s success
  • The AlexNet moment and technical innovations
Papers
  • ImageNet Classification with Deep Convolutional Neural Networks (2012) Krizhevsky, Sutskever, and Hinton

In 2012, Ilya, alongside Alex Krizhevsky and Geoffrey Hinton, trained a convolutional neural network to classify images. Their network stunned the AI community by dramatically reducing error rates in the emerging ImageNet Large Scale Visual Recognition Challenge (ILSVRC). AlexNet achieved a top-5 error rate of 15%, significantly outperforming conventional methods reliant on handcrafted feature engineering, which had an error rate of around 26%. The top-5 error rate measures how often the correct answer is not among the model’s top five guesses.

2.1 Feature engineering

2.2 Pre-AlexNet skepticism

2.3 ImageNet

2.4 Why training is hard

2.5 AlexNet

2.5.1 Network architecture

2.5.2 Training innovations

2.5.3 Data augmentation

2.5.4 Efficient and scalable training

2.5.5 The effect