Token-F1 measures overlap between a generated answer and a reference. It combines token-level precision and recall through their harmonic mean, producing a score from 0 to 1. Unlike exact match, it rewards partial overlap rather than requiring the whole answer to match. section 6.3.1 section 5.1.4
Use token-F1 to compare models on the same held-out questions and references, reporting both overall and per-category scores. For reproducible comparisons, an evaluation can use greedy decoding; production generation may instead use sampling. section 6.3.2 section 7.2.2
A small gain with a safety tradeoff
On 50 held-out IT-support examples, a LoRA adapter scored 0.154 token-F1 versus 0.149 for the base model. Safety refusal fell from 100% to 60%. The small overlap gain alone does not justify overlooking that safety regression. section 5.1.3 section 5.1.4
Score bands depend on the task. For short extractive answers, 0.20–0.35 indicates weak instruction-following, 0.35–0.55 moderate alignment, and 0.60 or higher strong alignment with the reference style. Long, open-ended answers can score much lower, so these bands should not be transferred to them. section 5.1.4
Overlap is not correctness
A useful paraphrase of VPN setup instructions scores 0.31, while an answer that shares more vocabulary but adds a nonexistent UI instruction scores 0.83. Token-F1 can miss equivalent wording and reward surface matching despite an error. section 7.2.3
Pair token-F1 with checks that address what overlap misses:
- Use exact match for strict agreement, but expect it to be unforgiving when answers are rephrased. section 5.1.4
- Check task-specific behavior, such as whether an answer names the correct internal tool and follows the required format. section 5.1.8
- Check safety separately and review responses qualitatively. For a final distillation candidate, an LLM judge can assess correctness, completeness, and coherence alongside token-F1. section 6.3.2 section 7.2.3
