liveBook · Manning

Greedy decoding

As explained in LLM Customization and Fine-Tuning

Greedy decoding is a generation method that selects the highest-probability next token at each step.

Greedy decoding selects the highest-probability token at each generation step. It makes model comparisons deterministic, so score differences are not obscured by sampling noise. section 7.2.1

Separate training-data diversity from evaluation

Teacher outputs are generated with temperature 0.7 for diversity, while evaluation uses greedy decoding for reproducible comparisons across base, teacher, and student models. section 7.2.1

Listing 7.4 Evaluating distillation quality
# chapter07/eval_distillation.py (key sections -- full source in repo)
 
from chapter05.metrics import token_f1                         #A
 
def evaluate_model(model, tokenizer, examples, label: str):
    """Run evaluation and return per-category token-F1."""
    cat_f1 = defaultdict(list)
    for ex in examples:
        msgs = ex["messages"]
        prompt = next(m["content"] for m in msgs if m["role"] == "user")
        reference = next(m["content"] for m in msgs if m["role"] == "assistant")
        category = ex.get("category", "unknown")
 
        generated = generate_response(model, tokenizer, prompt)
        cat_f1[category].append(token_f1(generated, reference)) #B
 
    f1_by_cat = {
        cat: sum(scores) / len(scores)
        for cat, scores in sorted(cat_f1.items())
    }
    overall_f1 = (
        sum(s for scores in cat_f1.values() for s in scores)
        / max(sum(len(s) for s in cat_f1.values()), 1)
    )
    return f1_by_cat, overall_f1
 
# In main():
# 1. Evaluate base model
# 2. Evaluate teacher (Ch6 SFT model)
# 3. Evaluate student (LoRA adapter on base)
model = AutoModelForCausalLM.from_pretrained(                  #C
    args.base_model, torch_dtype=torch.bfloat16, device_map="auto", ...
)
model = PeftModel.from_pretrained(model, args.student_dir)

See the evaluation listing that applies the same deterministic generation setup across models and scores responses with Token-F1. section 7.2.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines highest-probability token selection at each step and explains its role in reproducible evaluation.
Examples
Contrasts temperature-0.7 teacher-data generation with greedy evaluation of base, teacher, and student models.
What you can do after
Evaluate model responses reproducibly by selecting the highest-probability token at each generation step.
How it connects
Connects greedy decoding to sampling, knowledge distillation, and Token-F1 evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.