9 Error analysis
Once we’ve assembled the initial building blocks, which include gathering the first dataset, picking the metrics, defining the evaluation procedure, and training the baseline, we are ready to start the iterative adjustments process. Just as backpropagation in neural networks calculates the direction of the fastest loss reduction and passes it backward from layer to layer, error analysis finds the fastest improvement for the whole system.
Error analysis steps up as the compass that guides the iterative updates of your system. It helps you understand the error dynamics during the training phase (learning curve analysis) and the distribution of errors after the prediction phase (residual analysis). By analyzing these errors, you can identify commonalities, trends, and patterns that inform improvements to your machine learning (ML) system. In this chapter, we will examine its crucial stages and types and provide examples that we hope will give you a better understanding of the subject.
Error analysis is often skipped when ML systems are designed for a reason that seems somewhat legit at first glance—this step is not part of building the system per se. However, time spent on error analysis is always a good investment as it reveals weak spots and suggests ways of improving the system. Leaving this step out of this book would be a huge mistake from our side.