Verification checks teacher-generated data before training: outputs can be incorrect, unhelpful, repetitive, truncated, or refusals to benign prompts. Use automated filters first, then manually spot-check a sample for subtler problems such as factual errors or unhelpful answers. section 7.1.3
Use a two-layer review and track the results:
- Filter for response length, repetition, and format; these checks catch common failures but do not establish correctness. section 7.1.3
- Have a domain expert inspect a random 10% sample and record its error rate. If more than 20% of the sample has quality issues, adjust the prompts or teacher configuration before proceeding. section 7.1.3
- If domain experts are unavailable, use a different model to review a sample. Treat this as a lower-cost supplement, not a replacement for human review in the initial iteration. section 7.1.3
- Check for overlap with the held-out evaluation set and remove training examples whose similarity to a test example exceeds 0.85. section 7.1.3
Cross-model review
For example, use Claude Opus 5 to review a sample of GPT-6 Astra outputs for correctness and completeness. Cross-model review can catch some errors at lower cost than human review, but should supplement initial human spot-checking. section 7.1.3
Listing 7.2 Preparing distillation data
# Key sections of chapter07/scripts/prepare_distillation_data.py
def quality_filter(example: dict, min_words: int, max_words: int) -> bool:
"""Basic quality filter on teacher responses."""
msgs = example["messages"]
assistant_msg = next(
(m["content"] for m in msgs if m["role"] == "assistant"), ""
)
word_count = len(assistant_msg.split())
if word_count < min_words: #A
return False
if word_count > max_words: #B
return False
# Check for degenerate repetition
sentences = assistant_msg.split(".")
if len(sentences) > 3:
unique_sentences = {s.strip().lower() for s in sentences if s.strip()}
if len(unique_sentences) < len(sentences) * 0.5: #C
return False
return True
# In main():
filtered = [
ex for ex in examples
if quality_filter(ex, args.min_response_words, args.max_response_words)
]
# ... shuffle, split, write manifestThe quality-filter example checks response length and repeated sentences before preparing distillation data. section 7.1.3
from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity # Drop any training example too similar to a held-out test example. vec = TfidfVectorizer().fit(train_texts + test_texts) sims = cosine_similarity(vec.transform(train_texts), vec.transform(test_texts)) clean = [ex for ex, s in zip(train_examples, sims) if s.max() <= 0.85]
The similarity example removes training examples that exceed the stated threshold against held-out test examples. section 7.1.3
Verification matters because a student can faithfully reproduce a teacher’s consistent errors as well as its consistent behavior. section 7.4.3








