liveBook · Manning

Confidence interval

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A confidence interval is a range calculated from sample data that expresses uncertainty about a population parameter under a specified confidence procedure.

A confidence interval is an uncertainty range around an estimate. It shows how much trust to place in a reported metric instead of treating its point estimate as exact. section 4.1.5 section 4.1.2

Use confidence intervals to make evaluation results more actionable:

  • Report the interval alongside the estimate; a correlation without uncertainty bounds is incomplete. section 4.1.5
  • Interpret interval width diagnostically: a wide interval may mean the sample is too small for a reliable decision. section 4.1.5 section 4.1.2
  • Compare effect sizes and intervals rather than relying on binary p-value significance. section 4.1.5

Bootstrap confidence interval for Spearman’s ρ

Compute Spearman’s correlation, repeatedly resample paired observations with replacement, recompute the correlation, and use the 2.5th and 97.5th percentiles as the 95% interval. With only 20 samples, an interval from 0.4 to 0.9 can be too wide for confident decisions; larger samples produce more precise estimates. section 4.1.5 section 4.1.2

Listing 4.1 Computing Spearman's correlation with confidence intervals
import numpy as np
from scipy.stats import spearmanr
 
human_scores = [4, 2, 5, 3, 1, 4, 3, 5, 2, 1]
metric_scores = [0.7, 0.3, 0.9, 0.5, 0.1, 0.8, 0.4, 0.85, 0.25, 0.15]
 
corr, p_value = spearmanr(human_scores, metric_scores) #A
 
n_bootstraps = 1000
bootstrap_corrs = []
for _ in range(n_bootstraps): #B
    indices = np.random.choice(len(human_scores), len(human_scores), replace=True)
    resampled_human = [human_scores[i] for i in indices]
    resampled_metric = [metric_scores[i] for i in indices]
    boot_corr, _ = spearmanr(resampled_human, resampled_metric)
    bootstrap_corrs.append(boot_corr)
 
lower = np.percentile(bootstrap_corrs, 2.5) #C
upper = np.percentile(bootstrap_corrs, 97.5)
print(f"Spearman's ρ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
 

Use the runnable NumPy and SciPy listing to compute Spearman’s ρ, bootstrap resamples, and percentile-based confidence bounds. section 4.1.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains interval interpretation and bootstrap computation, while deferring formal statistical derivations.
Examples
Uses sample-size scenarios and Python code that bootstraps Spearman’s correlation from paired human and metric scores.
What it assumes
Assumes familiarity with correlations, point estimates, and basic resampling procedures.
What you can do after
Evaluate metric reliability by reporting estimates with uncertainty bounds and deciding when additional data is needed.
How it connects
Connects confidence intervals to bootstrap resampling, Spearman’s rank correlation, effect sizes, p-values, and Cohen’s kappa.

Also explained in

Compare how 12 books explain confidence interval →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.