liveBook · Manning

F-score

As explained in LLM Evaluation and Alignment, The Foundational Ideas

The F-measure combines precision and recall using their harmonic mean so that a high value requires both to be strong.

A system can score well on one objective while failing another. The F-score combines precision and recall using their harmonic mean, so a high score requires both to be strong. In binary classification, precision reflects avoiding false positives, while recall reflects capturing positive instances. section 2.2.2 section 2.3.2

For precision P and recall R, F1 = 2PR / (P + R). The harmonic mean is pulled toward the lower value: an excellent precision score cannot fully compensate for poor recall, or vice versa. Unlike an arithmetic average, it penalizes imbalance sharply. section 2.2.2 section 2.3.2

Summarization extremes

A summary that includes the entire source document may achieve perfect recall but suffer in precision. A one-word summary may achieve perfect precision but fail in recall. An F-score penalizes both extremes. section 2.2.2

The F-measure can also weight the objectives differently. A β of 1 gives equal weight to precision and recall; β greater than 1 emphasizes recall, while β below 1 emphasizes precision. In ROUGE-L, these scores are based on the longest common subsequence: matching words must preserve their relative order, but can have gaps. section 2.2.2

Figure 2.5 Impact of mean selection on imbalanced objectives. When evaluating a system with high helpfulness (0.90) but low safety (0.10), the choice of mean determines the pass/fail outcome. The arithmetic mean suggests acceptable performance (0.50). The geometric mean imposes a moderate penalty (0.30). The harmonic mean (0.18) enforces the strictest penalty for the imbalance, ensuring safety cannot be compromised for helpfulness.

The same harmonic-mean principle can balance other competing objectives. Here, high helpfulness and low safety produce a much lower harmonic mean than arithmetic mean. section 2.3.2

An F-score is only as informative as the quantities it combines. In ROUGE, lexical matching cannot reliably assess meaning, synonyms, or coherence. Use complementary measures and, where needed, human evaluation rather than treating a single score as overall quality. section 2.2.3 section 2.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the harmonic mean, explains its penalty for imbalance, and describes how β changes precision-recall weighting.
Examples
Contrasts full-document and one-word summaries, and compares means for helpfulness and safety scores.
What you can do after
Evaluate precision-recall tradeoffs and recognize when a harmonic mean exposes an imbalance.
How it connects
Connects F-scores to precision, recall, ROUGE-L, longest common subsequence, and harmonic means.

Also explained in

Compare how 20 books explain f-score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.