7 Validation schemas
This chapter covers
- Ensuring reliable evaluation
- Standard validation schemas
- Nontrivial validation schemas
- Split updating procedure
- Validation schemas as part of the design document
Building a robust evaluation process is essential for a machine learning (ML) system, and in this chapter, we will cover the process of building a proper validation schema to achieve confident estimates of system performance. We will touch upon typical validation schemas, as well as how to select the right validation based on the specifics of a given problem and what factors to consider when designing the evaluation process in the wild.
A proper validation procedure aims to imitate what knowledge we are supposed to have and what knowledge can be dropped while operating in a real-life environment. This is somewhat connected to the overfitting problem or generalization, which we’ll cover in detail in chapter 9.
It also provides a reliable and robust estimation of a system’s performance, ideally with some theoretical guarantees. As an example, we guarantee that a real value will be in the range between the lower confidence bound and upper confidence bound 95 times out of 100 (this case will be covered in a campfire story from Valerii later in the chapter). It also helps detect and prevent data leaks, overfitting, and divergence between offline and online performance.