7 Working with the People Annotating your Data
This chapter covers
- Understanding the characteristics of in-house, contracted, and pay-per-task annotation workforces.
- Motivating different workforces using three key principles.
- Evaluating workforces when compensation is non-monetary.
- Evaluating your annotation volume requirements.
- Understanding the training and/or expertise that annotators need for a given task.
In the last few chapters of the book, you learned how to select the right data for human review. Now, the following chapters will cover how to optimize that human interaction. Machine Learning models often require thousands (and sometimes millions) of instances of human feedback in order to get the training data necessary to be accurate.
The type of workforce you need will depend on your task, scale, and urgency. If you have a simple task, like identifying whether a social media posting is positive or negative sentiment and you need millions of human annotations as soon as possible, then your ideal workforce doesn’t need specialized skills. But ideally, that workforce can scale to thousands of people in parallel and each person can be employed for short amounts of time.