The final two chapters complete the loop with a deep dive on interfaces for effective annotation and three examples of human-in-the-loop machine learning applications. The chapters bring together everything you have learned in the book so far, showing how the interface design strategies are influenced by your data sampling and annotation strategies. The most optimal systems are designed holistically with all components in mind.
Chapter 11 shows how human–computer interaction principles can be applied to annotation interfaces and how different types of interfaces can automate some of the annotation process. The chapter covers the nontrivial trade-offs in interface design among annotation efficiency, annotation quality, agency of the annotators, and the engineering effort required to implement each type of interface.
Chapter 12 briefly discusses how to define products for human-in-the-loop machine learning applications and then walks through three example implementations: a system for exploratory data analysis for short text, a system to extract information from text, and a system to maximize the accuracy for an image labeling task. For each example, some potential extensions from other strategies in this book are listed, which will help you critically evaluate how to extend human-in-the-loop machine learning systems after you deploy your first applications.