5 Loss functions and metrics

 

This chapter covers

  • Selecting proper metrics and losses for your machine learning system
  • Defining and utilizing proxy metrics
  • Applying the hierarchy of metrics

In the previous chapter, we first touched on the topic of creating a design document for your machine learning (ML) system. We figured out why a design document is subject to constant edits and why all the changes you implement in it are not only inevitable but also necessary.

Unfortunately, an ML system can’t directly solve a problem, but it can try to approximate it by optimizing a specific task. To do that efficiently, it must be adjusted, appropriately guided, and monitored.

To direct an ML system’s effort, we use its algorithm’s loss function to reward or punish if for reducing or increasing specific errors. However, the loss function is used to train the model and usually must be differentiable, meaning that there is a narrowed choice of available loss functions. Thus, to assess the model’s performance, we use metrics; and while every loss function can be used as a metric (a good example would be root mean squared error [RMSE], which is quite often used as a metric, although we are not sure that is the best decision), not every metric can be used as a loss function.

In this chapter, we will discuss how to pick the best-fitting metrics and loss functions, focusing on how to do proper research and provide motivation for choice during the design process.

5.1 Losses

5.1.1 Loss tricks for deep learning models

5.2 Metrics

5.2.1 Consistency metrics

5.2.2 Offline and online metrics, proxy metrics, and hierarchy of metrics

5.3 Design document: Adding losses and metrics

5.3.1 Metrics and loss functions for Supermegaretail

5.3.2 Metrics and loss functions for PhotoStock Inc.

5.3.3 Wrap up

Summary