A performance metric is a numerical measure used to judge how well a model is doing. For a classifier, that might be accuracy; for a price predictor, it might be mean absolute error (MAE). Choose a metric that reflects success on the task, then monitor it during training and validation. The loss has a different job: it is the quantity training directly minimizes. A metric need not be differentiable. section 3.6.4 section 4.3.4
Check classification accuracy
In R with Keras, create metric_sparse_categorical_accuracy(), call update_state(targets, predictions), and read result(). In the supplied example, all three predictions match their targets, so result() is 1.00. In a custom loop, reset_state() at the start of a new epoch or evaluation so the next result does not include earlier batches. section 7.4.3
metric <- metric_sparse_categorical_accuracy()
targets <- op_array(c(0, 1, 2), dtype = "int32")
predictions <- op_array(rbind(c(1, 0, 0), c(0, 1, 0), c(0, 0, 1)))
metric$update_state(targets, predictions)
current_result <- metric$result()
cat(sprintf("result: %.2f\n", current_result))result: 1.00
The runnable example shows the metric receiving targets and predictions, accumulating its state, and returning an accuracy of 1.00. section 7.4.3
For a built-in training loop, pass metrics to compile() alongside the optimizer and loss, then train with fit(). Keras accepts metric shortcuts or metric objects; objects also allow further configuration or custom metrics. In a custom train_step(), update each compiled metric and return its result in the logs. The compiled metrics include a tracker for loss. section 3.6.4 section 7.4.5
Treat a validation score as an estimate, not a verdict. With few data points, changing which examples go into validation can change the score substantially. K-fold cross-validation trains and evaluates across several partitions, then averages their scores. In a price-prediction example, four validation MAE scores range from 0.266 to 0.336; their mean is about 0.29. Plotting mean validation MAE by epoch also helps identify when additional training stops improving it significantly. section 4.3.4
Listing 4.25 K-fold validation
k <- 4
fold_id <- sample(rep(1:k, length.out = nrow(train_data)))
num_epochs <- 50
all_scores <- numeric(k)
for (i in 1:k) {
cat(sprintf("Processing fold #%i\n", i))
fold_val_indices <- which(fold_id == i)
fold_x_val <- x_train[fold_val_indices, ] #1
fold_y_val <- y_train[fold_val_indices]
fold_x_train <- x_train[-fold_val_indices, ] #2
fold_y_train <- y_train[-fold_val_indices] #2
model <- get_model() #3
model |> fit( #4
fold_x_train, fold_y_train,
epochs = num_epochs, batch_size = 16, verbose = 0
)
results <- model |> evaluate(fold_x_val, fold_y_val, verbose = 0) #5
all_scores[i] <- results$mean_absolute_error
}
Processing fold #1
Processing fold #2
Processing fold #3
Processing fold #4
The R loop evaluates a fresh model on each held-out fold and records its mean absolute error; the fold scores can then be averaged. section 4.3.4
Decide how to measure success before refining a model. Use validation performance to tune it, keep a separate test set for the final check, and continue monitoring after deployment. Production checks can include fresh labeled predictions and business outcomes: an accurate recommender, for example, should still be checked for its effect on user engagement. Randomized A/B testing can help separate that effect from other changes. section 20.1.5 section 6.3.3
