5 Loss functions and metrics
This chapter covers
- Selecting proper metrics and losses for your ML system
- Defining and utilizing proxy metrics
- Applying the hierarchy of metrics
In the previous chapter, we first touched on the topic of creating a design document for your ML system. We figured out why a design document is subject to constant edits and why all the changes you implement to it are not only inevitable, but also necessary.
Unfortunately, a machine learning system can't directly solve a problem but can try to approximate it through optimizing a specific task. To do that efficiently, it must be adjusted, appropriately guided, and monitored.
To direct a machine learning system’s effort, we use its algorithm's loss function, to reward or punish for reducing or increasing specific errors. However, the loss function is used to train the model and usually must be differentiable, meaning that there is a narrowed choice of available loss functions. Thus, to assess the model’s performance, we use metrics, and while every loss function can be used as a metric (a good example would be Root Mean Squared Error quite often used as a metric, though we are not sure if it is the best decision), not every metric can be used as a loss function.
In this chapter, we will discuss how to pick best fitting metrics and loss functions, focusing on how to do proper research and provide motivation for choice during the design process.