15 Quality and acceptance testing

This chapter covers

Establishing consistency for data sources used in ML
Handling prediction failures gracefully with fallback logic
Providing quality assurance for ML predictions
Implementing explainable solutions

In the preceding chapter, we focused on broad and foundational technical topics for successful ML project work. Following from those foundations, a critical infrastructure of monitoring and validation needs to be built to ensure the continued health and relevance of any project. This chapter focuses on these ancillary processes and infrastructure tooling that enable not only more efficient development, but easier maintenance of the project once it is in production.

Between the completion of model development and the release of a project are four main activities:

Data availability and consistency verifications
Cold-start (fallback or default) logic development
User-acceptance testing (subjective quality assurance)
Solution interpretability (explainable AI, or XAI)

To show where these elements fit within a project’s development path, figure 15.1 illustrates the post-modeling phase work covered in this chapter.

Figure 15.1 Production-grade qualification and testing phase for an ML project

These highlighted actions are generally seen as an afterthought or reactive implementation for many projects that I’ve had exposure to. While not applicable to every ML solution, evaluating each of these components is highly recommended.

15 Quality and acceptance testing

This chapter covers

Figure 15.1 Production-grade qualification and testing phase for an ML project

15.1 Data consistency

15.1.1 Training and inference skew

15.1.2 A brief intro to feature stores

15.1.3 Process over technology

15.1.4 The dangers of a data silo

15.2 Fallbacks and cold starts

15.2.1 Leaning heavily on prior art