Spot-checking uses human review of a sample to catch quality problems automated checks miss, including factual errors, inconsistent reasoning, and technically correct but unhelpful answers. section 7.1.3
Use spot-checking as a quality gate:
- After automated filtering, have a domain expert review a random 10% sample and record the error rate as a data-quality metric. section 7.1.3
- If more than 20% of the sample has quality issues, adjust the prompts or teacher configuration before proceeding. section 7.1.3
- When expert review is unavailable, use a different model to review outputs as a lower-cost supplement, not a replacement for human review in the initial iteration. section 7.1.3
Before shipping a DPO model
Generate 20 to 30 responses to representative production prompts and have a domain expert review them. Metrics show aggregate trends; spot checks reveal individual failure modes. section 8.2.9
