Automated filters remove teacher-generated distillation examples that fail basic checks before student training. They catch common mechanical failures, not whether an answer is factually sound or useful. section 7.1.3 section 7.1
Common checks target:
- Responses that are too short or too long. section 7.1.3
- Repeated sentences and other degenerate outputs. section 7.1.3
- Invalid output formats. section 7.1.3
Listing 7.2 Preparing distillation data
# Key sections of chapter07/scripts/prepare_distillation_data.py
def quality_filter(example: dict, min_words: int, max_words: int) -> bool:
"""Basic quality filter on teacher responses."""
msgs = example["messages"]
assistant_msg = next(
(m["content"] for m in msgs if m["role"] == "assistant"), ""
)
word_count = len(assistant_msg.split())
if word_count < min_words: #A
return False
if word_count > max_words: #B
return False
# Check for degenerate repetition
sentences = assistant_msg.split(".")
if len(sentences) > 3:
unique_sentences = {s.strip().lower() for s in sentences if s.strip()}
if len(unique_sentences) < len(sentences) * 0.5: #C
return False
return True
# In main():
filtered = [
ex for ex in examples
if quality_filter(ex, args.min_response_words, args.max_response_words)
]
# ... shuffle, split, write manifestThis Python filter applies word-count bounds and rejects responses with excessive repeated sentences. section 7.1.3
Spot-check threshold
Review a random 10% sample. If more than 20% of that sample has quality issues, adjust the prompts or teacher configuration before proceeding. section 7.1.3
