6 Regularization via Model
This chapter covers
- Inductive bias and its use in convolutional neural networks
- Applying dropout as a regularization technique to improve model performance
- Implicit regularization using multi-task learning
Suitable regularization, whether explicit or implicit, is essential to achieving good generalization performance when using over-parameterized models such as deep neural networks. As introduced in the previous chapter, data augmentation promises a performance boost in visual recognition tasks such as image classification. However, generating valuable artificial datasets using the available training dataset still depends on a good coverage of the population data in terms of data variety. There is not much we can augment when the training set is extremely limited.
Consider, for example, the case of a minimal training set in figure 6.1, which is also the exact figure we encountered in chapter 1. The true underlying function is a curve and is unavailable to us. In this scenario, we would avoid unnecessary complexity in the model as it will easily overfit the training dataset consisting of six data points. The same reasoning applies when the training data is even more reduced. In extreme uncertainties, it is better to remain conservative and avoid undue complexity.
Figure 6.1 Presenting the same scenario with only six data points available. A simple linear regression model is preferred over an overly complex model such as deep neural networks.
