Statistical power is the ability of a small model to beat a simple baseline on the task. It is the first practical milestone: evidence that the features, architecture, and training setup are pointing in the right direction. section 6.2.3
To achieve statistical power, focus on:
- Filtering uninformative features and creating useful ones through feature engineering. section 6.2.3
- Choosing suitable architecture priors, such as densely connected, convolutional, recurrent, or Transformer models. section 6.2.3
- Selecting a workable loss function, batch size, and learning rate; optimize a differentiable proxy when the success metric cannot be optimized directly. section 6.2.3
A first model on MNIST
A logistic regression model can show statistical power on MNIST by beating a simple baseline, even though it may lack enough capacity to solve the task well. After this milestone, enlarge the model until it overfits, then regularize and tune it for generalization. section 6.2.4 section 6.2.5
Table 6.1 Which loss function, last-layer activation, and metrics to use for different tasks
|
Task
|
Last-layer activation
|
Loss function
|
Metrics
|
|---|---|---|---|
|
Binary classification
|
Sigmoid
|
Binary cross-entropy
|
Binary accuracy, ROC AUC
|
|
Multiclass, single-label classification
|
Softmax
|
Categorical cross-entropy
|
Categorical accuracy, top-
k categorical accuracy, ROC AUC
|
|
Multiclass, multilabel classification
|
Sigmoid
|
Binary cross-entropy
|
Binary accuracy, ROC AUC
|
|
Regression
|
None
|
Mean squared error
|
Mean absolute error
|
Use the task table to match common problem types with last-layer activations, loss functions, and evaluation metrics. section 6.2.3
