liveBook · Manning

Harmonic mean

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A harmonic mean is an average based on the reciprocal values of its inputs that is strongly influenced by the smallest input.

The harmonic mean averages reciprocal values and is strongly influenced by the smallest input. It produces a high score only when all required objectives are strong, so one weak dimension cannot be hidden by another strong dimension. section 2.2.2

Use the harmonic mean when:

  • Precision and recall must both be high, as in the F-measure. section 2.2.2
  • Competing objectives such as helpfulness and safety cannot compensate for each other. section 2.2.2
  • A custom evaluation metric must penalize failures rather than smooth them away with an arithmetic average. section 2.3.2

Imbalanced helpfulness and safety

For helpfulness of 0.90 and safety of 0.10, the arithmetic mean is 0.50, the geometric mean is 0.30, and the harmonic mean is 0.18. The harmonic mean gives the strictest penalty, preventing high helpfulness from masking poor safety. section 2.3.2

Figure 2.5 Impact of mean selection on imbalanced objectives. When evaluating a system with high helpfulness (0.90) but low safety (0.10), the choice of mean determines the pass/fail outcome. The arithmetic mean suggests acceptable performance (0.50). The geometric mean imposes a moderate penalty (0.30). The harmonic mean (0.18) enforces the strictest penalty for the imbalance, ensuring safety cannot be compromised for helpfulness.

Compare arithmetic, geometric, and harmonic means to choose an aggregation rule that matches the cost of imbalanced system performance. section 2.3.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines the reciprocal-based average, explains its strict penalty, and compares it with arithmetic and geometric means.
Examples
Uses precision-recall evaluation and helpfulness-safety scores of 0.90 and 0.10 to demonstrate strict aggregation.
What you can do after
Evaluate competing objectives with an aggregation rule that exposes weak dimensions instead of averaging them away.
How it connects
Connects precision, recall, F-measure, arithmetic mean, geometric mean, BLEU, ROUGE, and AI evaluation.

Also explained in

Compare how 2 books explain harmonic mean →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.