4 Evaluation metrics for classification
This chapter covers
- Accuracy as a way of evaluating binary classification models and its limitations
- Determining where our model makes mistakes using a confusion table
- Deriving other metrics like precision and recall from the confusion table
- Using ROC (receiver operating characteristics) and AUC (area under the ROC curve) to further understand the performance of a binary classification model
- Cross-validating a model to make sure it behaves optimally
- Tuning the parameters of a model to achieve the best predictive performance
In this chapter we will continue with the project we started in the previous chapter: churn prediction. We have already downloaded the dataset, performed the initial preprocessing, exploratory data analysis, and trained the model that predicts whether customers will churn. We have also evaluated this model on the validation dataset and concluded that it has 80% accuracy.
The question we postponed until now was whether 80% accuracy is good or not and what it actually means in terms of the quality of our model. We will answer this question in this chapter and discuss other ways of evaluating a binary classification model: confusion table, precision and recall, the ROC curve and AUC.