A labeling platform organizes data annotation around tasks, instructions, interfaces, workers, quality control, aggregation, and pricing. Choose it after clarifying product goals and the detail labels require. section 6.2.4
Evaluate a labeling setup by checking:
- Whether tasks are decomposed clearly and instructions are easy to follow. section 6.2.4
- Whether interfaces, templates, worker training, examination, and process evaluation support reliable labels. section 6.2.4
- Whether in-house experts or third-party crowdsourcing better matches labeling complexity, control, consistency, speed, and cost. section 6.2.4
- Whether weak supervision is preferable when detailed manual labeling is too slow, costly, or expertise-constrained. section 6.2.4
Choosing the workforce
Use a curated in-house team when annotations require specialized knowledge; use an external service when tasks are simpler and rapid scale or lower cost matters. section 6.2.4
Figure 6.2 Various approaches to data labeling
Compare the available approaches to data labeling before selecting a platform or workforce. section 6.2.4
