chapter six
6 Common Design Building Blocks
This chapter covers
- Adding new activation functions.
- Inserting new layers to improve training.
- Skipping layers as a useful design pattern.
- Combining new activations, layers, and skips into new approaches more powerful and the sum of their parts.
At this point we have learned about the three most common and fundamental types of neural networks: fully-connected, convolutional, and recurrent. We have been able to improve all of these architectures by changing the optimizer and learning rate schedule, which alter how we update the parameters (weights) of our models, giving us more accurate models almost “for free”. All of the things we have learned thus far also have a long shelf life, and have taught us about problems that have (and continue) existed for decades. This gives you a good foundation to “speak the language” of deep learning, and some very fundamental building blocks larger algorithms are made from.