chapter seven
7 Selecting the Right People to Annotate your Data
This chapter covers
- Understanding the characteristics of the three main types of annotation workforces: in-house, contracted, and pay-per-task.
- Applying the three key principles for motivating an annotator - salary, security & transparency - and how to apply the principles to different workforces.
- Evaluating workforces with non-monetary compensation systems, including application end-users, volunteers, people playing games, and computer-generated data/annotations.
- Evaluating your annotation volume requirements, in order to understand the scale and frequency of the annotations that you will need.
- Deciding on the qualifications necessary for an annotation task, in order to understand the amount of training and/or expertise needed by annotators.
In the last few chapters of the book, you learned how to select the right data for human review. Now, the following chapters will cover how to optimize that human interaction. Machine Learning models often require thousands (and sometimes millions) of instances of human feedback in order to get the training data necessary to be accurate.