A quality filter screens training examples before they reach a model. In knowledge distillation, it removes teacher responses that could teach a student terse answers or repetitive filler. The tradeoff is direct: an aggressive filter discards useful data, while a lenient one lets poor examples through. section 7.1.6 section 7.7.2
A basic filter checks response length and degenerate repetition. Test it with a normal response, responses that are too short and too long, and a repetitive response. Include these tests in CI and update them when domain-specific thresholds change. section 7.1.6 section 7.7.2
Calibrate against the teacher and domain rather than treating a rejection rate as a universal target:
- Start by examining a small batch. Dropping 10%–25% of outputs is a workable starting point for the described frontier-teacher setup, but a clean local teacher lost only 1 of 200 examples. section 7.1.6
- After each rule change, manually review 20 rejected outputs. Loosen the filter if most are reasonable answers; a high rejection rate can be appropriate if the rejects are mostly incomplete, repetitive, or refusals on benign prompts. section 7.1.6
- Use domain-expert spot checks and adjust thresholds during refinement, then retrain and compare with the baseline. section 7.6.6
Clinical training data
A clinical fine-tuning dataset used deduplication, length filtering, and manual review of a 200-example stratified sample. Filtering removed about 8% of the initial data, mainly near-duplicate questions and examples with incomplete reasoning or reasoning that contradicted the final answer. Early unfiltered experiments sometimes produced confident but internally contradictory reasoning. section 6.12.1 section 6.12.6
Length and repetition checks do not establish safety or appropriateness: a harmful answer can be fluent and the right length. Add safety-aware spot checks and evaluation. Do not indiscriminately remove refusals; refusals paired with harmful prompts can teach a student when to decline. section 7.8.1
