chapter four

4 Deep learning accelerates

 

This chapter covers

  • The resurgence of recurrent neural networks after AlexNet
  • How Karpathy’s blog made recurrent neural networks accessible, inspiring experimentation
  • How Chris Olah clarified long short-term memory with vivid visuals and metaphors
  • How selective dropout enabled deeper recurrent networks
  • How Deep Speech 2 proved the real-world potential of recurrent neural networks
  • The engineering shift in AI
Papers
  • The Unreasonable Effectiveness of Recurrent Neural Networks (2015) Andrej Karpathy
  • Understanding LSTM Networks (2015) Chris Olah
  • Recurrent Neural Network Regularization (2014) Zaremba, Sutskever, and Vinyals
  • Deep Speech 2: End-to-End Speech Recognition in English and Mandarin (2015) Amodei et al.

Recurrent neural networks (RNNs) are sequence models. Sequence models process ordered data such as text, speech, or time series, where the position of each element matters. n-grams are sequence models. They capture short-range dependencies by looking at a fixed window of prior tokens. RNNs extended this idea by carrying forward a compressed representation of the entire sequence history. Yet, despite their theoretical promise, practical problems such as vanishing or exploding gradients, poor handling of long-range temporal dependencies, and inefficient training limit their effectiveness. These challenges persisted until a dropout method was introduced, specifically designed for recurrent architectures.

4.1 The unreasonable effectiveness of RNNs

4.2 Understanding LSTM networks

4.3 Recurrent neural network regularization

4.4 Deep Speech 2

4.4.1 Core architecture

4.4.2 Training techniques

4.4.3 Language models and decoding

4.4.4 Significance and broader influence

4.4.5 An engineering shift