5 Loss functions and metrics
This chapter covers
- Selecting proper metrics and losses for your machine learning system
- Defining and utilizing proxy metrics
- Applying the hierarchy of metrics
In the previous chapter, we first touched on the topic of creating a design document for your machine learning (ML) system. We figured out why a design document is subject to constant edits and why all the changes you implement in it are not only inevitable but also necessary.
Unfortunately, an ML system can’t directly solve a problem, but it can try to approximate it by optimizing a specific task. To do that efficiently, it must be adjusted, appropriately guided, and monitored.
To direct an ML system’s effort, we use its algorithm’s loss function to reward or punish if for reducing or increasing specific errors. However, the loss function is used to train the model and usually must be differentiable, meaning that there is a narrowed choice of available loss functions. Thus, to assess the model’s performance, we use metrics; and while every loss function can be used as a metric (a good example would be root mean squared error [RMSE], which is quite often used as a metric, although we are not sure that is the best decision), not every metric can be used as a loss function.
In this chapter, we will discuss how to pick the best-fitting metrics and loss functions, focusing on how to do proper research and provide motivation for choice during the design process.