Data collection acquires preference data from external vendors to improve a model against a defined evaluation goal. Effective acquisition is an iterative process rather than a one-time purchase: teams refine goals, methods, annotator performance, and agreement while monitoring whether new data produces measurable gains. section 11.2.5
A practical collection operation must account for:
- Ramp-up batches that narrow the goals and methodology before full-scale collection. section 11.2.5
- Annotator performance and interannotator agreement as quality signals. section 11.2.5
- Discarding substantial amounts of early or low-quality data when it fails to support the target model improvement. section 11.2.5
Multibatch vendor collection
A smaller preference-data contract may cost about $500,000 and proceed through a ramp-up followed by successive batches. Earlier batches may be discarded at higher rates, while later batches reflect improved goals and methods. section 11.2.5
Figure 11.6 Overview of the multibatch cycle for obtaining human preference data from a vendor. The ramp-up period allows a narrowing of goals and methodology to create the best possible data. It is expected that a larger proportion of the data from the earlier batches will have to be thrown out due to quality issues. This is one timeline example for a smaller data contract (~$500,000); much larger data contracts can vary substantially.
Use the multibatch cycle to inspect how ramp-up, quality loss, and vendor data production fit together operationally. section 11.2.5
