liveBook · Manning

Geometric mean

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A geometric mean is the multiplicative average of a set of positive values, computed as their product raised to the reciprocal of their count.

A geometric mean combines values multiplicatively: multiply n values, then take the nth root. Unlike the arithmetic mean, which adds values before dividing by their count, it penalizes low components strongly. If any component is zero, the unsmoothed geometric mean is zero. section 2.1.3 section 2.1.4

Combining translation scores

BLEU combines precision scores for different n-gram lengths, typically one through four. For scores 0.80, 0.60, 0.40, and 0.20, the geometric mean is the fourth root of 0.0384, or approximately 0.442; the arithmetic mean is 0.50. If only the first score is nonzero, the geometric mean is zero, while the arithmetic mean is 0.20. BLEU then applies a brevity penalty to discourage overly short translations. section 2.1.3 section 2.4

This behavior is useful when every component must work. For example, a compound retrieval-augmented generation system can score retrieval, reasoning, and generation separately: a zero retrieval score makes the unsmoothed aggregate zero, even if generation scores highly. A moving geometric mean can also expose near-zero performance in a monitoring window rather than letting stronger results dilute it. section 2.3.2

Figure 2.4 Comparison of aggregation methods for a compound RAG system. The arithmetic mean (left) allows high scores in reasoning and generation to mask a near-total failure in retrieval (0.01), resulting in a deceptive passing score of 0.63. The geometric mean (right) propagates the retrieval failure, pulling the aggregate score down to 0.21 and correctly flagging the system error.

Compare how arithmetic and geometric aggregation respond to a near-failure in retrieval. section 2.3.2

For positive values, a numerically stable implementation takes the logarithm of each value, averages the logarithms, and exponentiates the result. This avoids underflow from multiplying many small values. Because log(0) is problematic, an implementation may replace zero with a small smoothing constant such as 1e-10; smoothing changes an exact zero aggregate into a small positive value. section 2.1.5

python
# Numerically stable calculation
epsilon = 1e-10                                                                     #A
precision_scores = [max(p, epsilon) for p in precisions]                            #B
geometric_mean = exp(sum(log(p) for p in precision_scores) / len(precision_scores)) #C

The Python listing computes the mean in log space and floors precision scores at a small epsilon. section 2.1.5

Choose the aggregation rule to match the failure you want to detect, and inspect the component scores as well as the aggregate. In BLEU, the mean combines lexical overlap scores; it does not by itself measure semantic equivalence or recognize paraphrases. section 2.1.4 section 2.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines and contrasts the mean, works through scores and stable implementation, and leaves the log-form proof as an exercise.
Examples
Calculates translation precision means and compares aggregation methods for retrieval, reasoning, and generation.
What you can do after
Evaluate and understand geometric aggregation for critical component scores, including zero handling and numerical stability.
How it connects
Connects geometric mean to arithmetic mean, n-gram precision, BLEU, brevity penalty, and retrieval-augmented generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.