Expressive power is the set of functions or behaviors a model architecture can represent. Increasing capacity lets a network fit more complex functions, including memorizing a dataset. section 6.2.7
Use these distinctions when diagnosing model behavior:
- A model that cannot fit randomly assigned labels may be too small; one that fits them has enough capacity to memorize the dataset. section 6.2.7
- Normalization layers do not increase representational capacity, but they improve what the model can learn and can help avoid a harder-to-reason-about training situation. section 6.2.7
Normalization without added power
A normalization layer follows a linear layer, yet algebra can rewrite the combined operations so the normalized form does not add representational power. Its value is improved learning rather than a larger function class. section 6.2.7
