A third-party crowdsourcing service supplies externally managed workers for manual annotation. It can scale labeling quickly and potentially reduce cost, but offers less control and consistency than an in-house team. section 6.2.4
Use an external service effectively by:
- Choosing it when tasks do not require specialized knowledge; use curated in-house experts for complex or skill-dependent labeling. section 6.2.4
- Decomposing tasks and writing simple, unambiguous instructions because labelers are interchangeable and may miss task nuances. section 6.2.4
- Adding easy-to-use interfaces, quality controls, performer training, aggregation, and pipeline evaluation because even simple labels can contain mistakes. section 6.2.4
Platform choice
Amazon Mechanical Turk is an example of a crowdsourcing platform; platform-specific features matter less than generic controls for instructions, interfaces, quality, and aggregation. section 6.2.4
Figure 6.2 Various approaches to data labeling
Compare external crowdsourcing with other data-labeling approaches before choosing a workforce model. section 6.2.4
