A useful dataset does not automatically have useful target values. Data annotation assigns labels to examples so they can serve a specific ML task. Labels may come from a dedicated expert team or from end users through an interface. Before organizing the work, decide what the product needs the model to determine—and what information a label must preserve. section 6.1 section 6.2.4
For a medical-image assistant, the annotation request could take several forms:
- Binary classification: Is a malignancy present? section 6.2.4
- Multiclass classification: What kind of malignancy is present, if any? section 6.2.4
- Detection: Which image region contains it? section 6.2.4
- Segmentation: Which pixels represent it? section 6.2.4
These are not interchangeable requests to the labeling team. They call for different tools, expertise, and amounts of time; the choice also affects which models and product behavior are possible. Start with the product goal rather than asking for the most detailed annotation by default. If the desired detail is impractical because expert time or tools are unavailable, consider whether inaccurate or partial labels and a weakly supervised approach are sufficient. section 6.2.4
Turn the decision into an annotation plan:
- Record which labels are needed and where they will come from. section 6.7
- Choose a platform; break work into tasks with readable instructions and usable interfaces. section 6.2.4
- Plan performer training and examination, quality control, label aggregation, evaluation, and pricing. section 6.2.4
Annotating image-search results
A stock-photo search system needs people to label query–image pairs as Relevant, Irrelevant, or Can’t answer. The team may later divide relevance into finer levels. Requiring a written reason for Can’t answer can reveal issues in early annotation rounds. The platform and staffing choice remains open: compare an in-house team with third-party services and get cost estimates before committing. section 6.7.2
Figure 6.2 Various approaches to data labeling
Compare the possible labeling approaches before choosing how much detail each annotated example must contain. section 6.2.4
