A labeling guideline is a documented set of instructions for annotators that defines task priorities and decision criteria. In preference-data collection, guidelines are refined during calibration so annotations become more consistent and useful for training. section 11.2.5 section 11.5
Effective guideline development is an iterative data-collection practice:
- Start with guidelines aligned to the training goal and the preference data required by the pipeline. section 11.2.5
- Use early batches to calibrate instructions, adjust the plan, remove underperforming annotators, and study interannotator agreement. section 11.2.5
- Check whether guideline-defined behaviors appear in trained models, because annotation choices can transfer biases into model behavior. section 11.5
Six-week preference-data contract
A six-week delivery period can reserve the first weeks for refining annotation guidelines and analyzing agreement, leaving later batches to improve the model after the process is calibrated. section 11.2.5
Figure 11.6 Overview of the multibatch cycle for obtaining human preference data from a vendor. The ramp-up period allows a narrowing of goals and methodology to create the best possible data. It is expected that a larger proportion of the data from the earlier batches will have to be thrown out due to quality issues. This is one timeline example for a smaller data contract (~$500,000); much larger data contracts can vary substantially.
The multibatch cycle visualizes ramp-up, narrowing goals and methodology, and the higher expected discard rate for early low-quality batches. section 11.2.5
