Classification accuracy is the fraction of predictions that match the true labels. It provides a direct score for comparing classification models, but its meaning depends on which data split produced it. section 5.4.2 section 13.3
Use accuracy with a disciplined evaluation workflow:
- Compute validation accuracy while searching for model or training hyperparameters. section 5.4.2
- Maximize the validation score during Optuna trials, then evaluate the selected model on the test set only afterward. section 5.4.2
- Compare accuracy across training strategies when the dataset size and evaluation procedure remain consistent. section 13.3
Comparing limited-data strategies
With very limited labeled data, training from scratch reached barely above 50% test accuracy, a warm start reached about 80%, and frozen pretrained convolutional layers reached about 91%. section 13.3
Figure 5.17 The four steps of defining an objective function that Optuna can use to optimize our neural network training. The two most important steps shown as code are getting the hyperparameters and computing the result!
Use the objective-function workflow to connect hyperparameter suggestions, model training, and the accuracy returned for optimization. section 5.4.2
