A metric is a quantitative measure of model performance or project success. Choose one that reflects the goal: accuracy and ROC AUC are common for balanced classification, while precision and recall can suit imbalanced problems. section 6.1.4
Keras provides built-in metrics for common classification and regression tasks. For an unusual measure, define a custom metric by subclassing Keras Metric and specifying how it tracks and reports its value. section 7.3.1
A stateful custom metric needs to:
- Update its internal state with update_state(); unlike layer variables, metric state is not updated through backpropagation. section 7.3.1
- Return the current value with result(), and clear accumulated state with reset_state() so the metric can be reused. section 7.3.1
Listing 7.18 Implementing a custom metric by subclassing the Metric class
metric_sparse_root_mean_squared_error <- new_metric_class( #1
classname = "SparseRootMeanSquaredError",
initialize = function(name = "rmse", ...) { #2
super$initialize(name = name, ...) #2
self$sum_sq_error <- self$add_weight( #2
name = "sum_sq_error", initializer = "zeros" #2
) #2
self$total_samples <- self$add_weight( #2
name = "total_samples", initializer = "zeros" #2
) #2
}, #2
update_state = function(y_true, y_pred, sample_weight = NULL) { #3
.[num_samples, num_classes] <- op_shape(y_pred) #3
y_true <- op_one_hot( #3
y_true, #3
zero_indexed = TRUE, #3
num_classes = num_classes #3
) #3
sse <- op_sum(op_square(y_true - y_pred)) #3
self$sum_sq_error$assign_add(sse) #3
self$total_samples$assign_add(num_samples) #3
}, #3
See a sparse-label RMSE metric implemented as a Keras Metric subclass, with internal state for accumulated squared error and sample count. section 7.3.1
Using a custom metric
Compile a model with both accuracy and a custom sparse-label RMSE metric, then use fit() to monitor training and validation and evaluate() to assess test data. section 7.3.1
