This chapter covers
- Calculating measurements of forecasting accuracy for churn
- Backtesting a model in a historical simulation
- Setting the regression parameter for the minimum metric contribution
- Picking the best value of the regression parameter by testing (cross-validation)
- Forecasting churn risk with the XGBoost machine learning model
- Setting the parameters of the XGBoost model with cross-validation
You know how to forecast the probability of customer churn, and you also know how to check the calibration of your forecasts. Another important measurement of a forecasting model is whether the customers predicted to be highly at risk are really more at risk than those predicted to be safe. This type of predictive performance is generally known as accuracy, although as you will see, there is more than one way to measure accuracy.
Back in chapter 1, I told you that forecasting churn with a predictive model was not the emphasis of this book because it isn’t helpful in many situations. The focus of this book is on having a good set of metrics that segment customers into healthy and unhealthy populations based on behavior. But there are a few reasons why it’s good to have accurate predictive churn forecasts, so this chapter will round out your skill set and ensure that you can forecast accurately when necessary.