Architecture priors are assumptions about which model structures fit a task. Choosing a structure that matches the data can make generalization possible; choosing a poor fit can leave a model unable to beat even a trivial baseline. section 5.3.2 section 6.2.3
Use architecture choice as an early modeling decision:
- Inspect the task and data modality before selecting a model. section 5.3.2
- Compare alternatives such as densely connected networks, ConvNets, recurrent neural networks, Transformers, or non-deep-learning methods. section 6.2.3
- Start with a small model and test whether it beats a simple baseline. section 6.2.3
- Research prior art for architectures and feature-engineering techniques that fit the problem. section 6.2.3
A temporal convolution can encode the wrong assumption
For hourly temperature forecasting, a 1D ConvNet can scan 24-hour windows and reuse representations across days. It performs poorly, however, because weather does not consistently satisfy translation invariance: morning, evening, and night have different properties. Pooling also discards important order information, while the recent past matters more than data from several days earlier. section 13.2.4
Table 6.1 Which loss, last-layer activation, and metrics to use for different tasks
| Task |
Last-layer activation |
Loss function |
Metrics |
|---|---|---|---|
| Binary classification |
Sigmoid |
Binary crossentropy |
Binary accuracy, ROC AUC |
| Multiclass, single-label classification |
Softmax |
Categorical crossentropy |
Categorical accuracy, top-k categorical accuracy, ROC AUC |
| Multiclass, multi-label classification |
Sigmoid |
Binary crossentropy |
Binary accuracy, ROC AUC |
| Regression |
None |
Mean squared error |
Mean absolute error |
Use the task-to-activation, loss, and metric table alongside architecture selection; it supplies practical starting templates for common problem types. section 6.2.3
