liveBook · Manning

P-value

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A p-value is the probability, under a specified null hypothesis, of obtaining a result at least as extreme as the observed result.

A p-value measures how likely an observed result, or a more extreme one, would be if the specified null hypothesis were true. For correlation evaluation, it tests whether the relationship could plausibly arise by chance. section 4.1.2

A small p-value supports evidence against a no-relationship null hypothesis, but it does not measure practical importance. Large samples can make tiny correlations statistically significant, so interpret p-values with effect sizes and confidence intervals. section 4.1.5 section 4.1.2

Interpret a p-value by checking three things:

  • Statistical evidence: p < 0.05 is presented as a common minimum, while p < 0.01 indicates stronger evidence. section 4.1.2
  • Effect size: a correlation such as ρ = 0.15 may remain practically weak despite p < 0.001. section 4.1.5
  • Uncertainty: report a confidence interval because the same point estimate can support different decisions depending on its precision. section 4.1.2

Comparing evaluation metrics

Suppose a metric agrees with human rankings at ρ = 0.15 and p < 0.001. The relationship is unlikely to be chance, but the metric barely predicts preferences. A different metric with ρ = 0.6 and p = 0.08 may deserve more data before rejection. section 4.1.5

Listing 4.1 Computing Spearman's correlation with confidence intervals
import numpy as np
from scipy.stats import spearmanr
 
human_scores = [4, 2, 5, 3, 1, 4, 3, 5, 2, 1]
metric_scores = [0.7, 0.3, 0.9, 0.5, 0.1, 0.8, 0.4, 0.85, 0.25, 0.15]
 
corr, p_value = spearmanr(human_scores, metric_scores) #A
 
n_bootstraps = 1000
bootstrap_corrs = []
for _ in range(n_bootstraps): #B
    indices = np.random.choice(len(human_scores), len(human_scores), replace=True)
    resampled_human = [human_scores[i] for i in indices]
    resampled_metric = [metric_scores[i] for i in indices]
    boot_corr, _ = spearmanr(resampled_human, resampled_metric)
    bootstrap_corrs.append(boot_corr)
 
lower = np.percentile(bootstrap_corrs, 2.5) #C
upper = np.percentile(bootstrap_corrs, 97.5)
print(f"Spearman's ρ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
 

Computes Spearman’s correlation and its p-value, then uses bootstrap resampling to report a 95% confidence interval. section 4.1.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains significance, computation, and interpretation, but stops short of treating significance as practical correlation strength.
Examples
Uses human and metric scores, bootstrap resampling, and contrasting p-values to illustrate evaluation decisions.
What it assumes
Assumes correlation results and null-hypothesis testing, while establishing statistical significance through evaluation examples.
What you can do after
Evaluate metric evidence by interpreting p-values alongside correlation magnitudes and confidence intervals.
How it connects
Connects p-values to Spearman’s rank correlation, confidence intervals, effect sizes, and statistical significance.

Also explained in

Compare how 9 books explain p-value →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.