Greedy decoding selects the highest-probability token at each generation step. It makes model comparisons deterministic, so score differences are not obscured by sampling noise. section 7.2.1
Separate training-data diversity from evaluation
Teacher outputs are generated with temperature 0.7 for diversity, while evaluation uses greedy decoding for reproducible comparisons across base, teacher, and student models. section 7.2.1
Listing 7.4 Evaluating distillation quality
# chapter07/eval_distillation.py (key sections -- full source in repo)
from chapter05.metrics import token_f1 #A
def evaluate_model(model, tokenizer, examples, label: str):
"""Run evaluation and return per-category token-F1."""
cat_f1 = defaultdict(list)
for ex in examples:
msgs = ex["messages"]
prompt = next(m["content"] for m in msgs if m["role"] == "user")
reference = next(m["content"] for m in msgs if m["role"] == "assistant")
category = ex.get("category", "unknown")
generated = generate_response(model, tokenizer, prompt)
cat_f1[category].append(token_f1(generated, reference)) #B
f1_by_cat = {
cat: sum(scores) / len(scores)
for cat, scores in sorted(cat_f1.items())
}
overall_f1 = (
sum(s for scores in cat_f1.values() for s in scores)
/ max(sum(len(s) for s in cat_f1.values()), 1)
)
return f1_by_cat, overall_f1
# In main():
# 1. Evaluate base model
# 2. Evaluate teacher (Ch6 SFT model)
# 3. Evaluate student (LoRA adapter on base)
model = AutoModelForCausalLM.from_pretrained( #C
args.base_model, torch_dtype=torch.bfloat16, device_map="auto", ...
)
model = PeftModel.from_pretrained(model, args.student_dir)See the evaluation listing that applies the same deterministic generation setup across models and scores responses with Token-F1. section 7.2.1
