liveBook · Manning

Statistical significance

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Statistical significance is the condition in which an observed result is sufficiently unlikely under a null hypothesis according to a chosen threshold.

Statistical significance means an observed result would be unlikely by chance if the null hypothesis were true. In evaluation, a p-value below 0.05 is commonly treated as significant, while 0.01 indicates stronger evidence. section 4.1.2

Interpret significance alongside uncertainty and practical magnitude:

  • Report confidence intervals with correlations; a point estimate without uncertainty can mislead. section 4.1.5
  • Treat p-values as evidence against chance, not as measures of practical importance. section 4.1.5
  • Check the magnitude of ρ, τ, or κ: large samples can make trivial correlations significant, while small samples can conceal meaningful agreement. section 4.1.5

A significant result is not necessarily useful

A metric with Spearman’s ρ = 0.15 and p < 0.001 is statistically significant, yet barely predicts human preferences. A metric with ρ = 0.6 and p = 0.08 may warrant more data rather than immediate rejection. section 4.1.5

Listing 4.1 Computing Spearman's correlation with confidence intervals
import numpy as np
from scipy.stats import spearmanr
 
human_scores = [4, 2, 5, 3, 1, 4, 3, 5, 2, 1]
metric_scores = [0.7, 0.3, 0.9, 0.5, 0.1, 0.8, 0.4, 0.85, 0.25, 0.15]
 
corr, p_value = spearmanr(human_scores, metric_scores) #A
 
n_bootstraps = 1000
bootstrap_corrs = []
for _ in range(n_bootstraps): #B
    indices = np.random.choice(len(human_scores), len(human_scores), replace=True)
    resampled_human = [human_scores[i] for i in indices]
    resampled_metric = [metric_scores[i] for i in indices]
    boot_corr, _ = spearmanr(resampled_human, resampled_metric)
    bootstrap_corrs.append(boot_corr)
 
lower = np.percentile(bootstrap_corrs, 2.5) #C
upper = np.percentile(bootstrap_corrs, 97.5)
print(f"Spearman's ρ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
 

Compute Spearman’s correlation, its p-value, and a bootstrap 95% confidence interval with Python and SciPy. section 4.1.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines p-values, thresholds, confidence intervals, and effect-size caveats, then stops short of broader hypothesis-testing procedures.
Examples
Contrasts small significant correlations with larger nonsignificant correlations and demonstrates bootstrap intervals for Spearman’s ρ.
What it assumes
Assumes familiarity with correlations and evaluation metrics while explaining significance, uncertainty, and sample-size effects directly.
What you can do after
Evaluate/understand whether an observed evaluation correlation supports a reliable claim, using p-values, effect sizes, and confidence intervals.
How it connects
Connects the null hypothesis, p-values, confidence intervals, effect sizes, and Spearman’s rank correlation.

Also explained in

Compare how 9 books explain statistical significance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.