liveBook · Manning

Token-F1

As explained in LLM Customization and Fine-Tuning

An F1 score based on token-level precision and recall between a predicted text and a reference text.

Token-F1 measures overlap between a generated answer and a reference. It combines token-level precision and recall through their harmonic mean, producing a score from 0 to 1. Unlike exact match, it rewards partial overlap rather than requiring the whole answer to match. section 6.3.1 section 5.1.4

Use token-F1 to compare models on the same held-out questions and references, reporting both overall and per-category scores. For reproducible comparisons, an evaluation can use greedy decoding; production generation may instead use sampling. section 6.3.2 section 7.2.2

A small gain with a safety tradeoff

On 50 held-out IT-support examples, a LoRA adapter scored 0.154 token-F1 versus 0.149 for the base model. Safety refusal fell from 100% to 60%. The small overlap gain alone does not justify overlooking that safety regression. section 5.1.3 section 5.1.4

Score bands depend on the task. For short extractive answers, 0.20–0.35 indicates weak instruction-following, 0.35–0.55 moderate alignment, and 0.60 or higher strong alignment with the reference style. Long, open-ended answers can score much lower, so these bands should not be transferred to them. section 5.1.4

Overlap is not correctness

A useful paraphrase of VPN setup instructions scores 0.31, while an answer that shares more vocabulary but adds a nonexistent UI instruction scores 0.83. Token-F1 can miss equivalent wording and reward surface matching despite an error. section 7.2.3

Pair token-F1 with checks that address what overlap misses:

  • Use exact match for strict agreement, but expect it to be unforgiving when answers are rephrased. section 5.1.4
  • Check task-specific behavior, such as whether an answer names the correct internal tool and follows the required format. section 5.1.8
  • Check safety separately and review responses qualitatively. For a final distillation candidate, an LLM judge can assess correctness, completeness, and coherence alongside token-F1. section 6.3.2 section 7.2.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines overlap-based F1 and demonstrates evaluation and interpretation, without detailing tokenization or implementing the metric itself.
Examples
Compares LoRA and base scores on IT-support answers and contrasts VPN paraphrases with a high-overlap incorrect instruction.
What you can do after
Evaluate model comparisons using token-F1 alongside safety and behavioral checks, while recognizing that overlap does not establish correctness.
How it connects
Connects token-F1 with exact match, safety refusal rate, behavioral evaluation, and LLM judges.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.