A confidence interval is an uncertainty range around an estimate. It shows how much trust to place in a reported metric instead of treating its point estimate as exact. section 4.1.5 section 4.1.2
Use confidence intervals to make evaluation results more actionable:
- Report the interval alongside the estimate; a correlation without uncertainty bounds is incomplete. section 4.1.5
- Interpret interval width diagnostically: a wide interval may mean the sample is too small for a reliable decision. section 4.1.5 section 4.1.2
- Compare effect sizes and intervals rather than relying on binary p-value significance. section 4.1.5
Bootstrap confidence interval for Spearman’s ρ
Compute Spearman’s correlation, repeatedly resample paired observations with replacement, recompute the correlation, and use the 2.5th and 97.5th percentiles as the 95% interval. With only 20 samples, an interval from 0.4 to 0.9 can be too wide for confident decisions; larger samples produce more precise estimates. section 4.1.5 section 4.1.2
Listing 4.1 Computing Spearman's correlation with confidence intervals
import numpy as np
from scipy.stats import spearmanr
human_scores = [4, 2, 5, 3, 1, 4, 3, 5, 2, 1]
metric_scores = [0.7, 0.3, 0.9, 0.5, 0.1, 0.8, 0.4, 0.85, 0.25, 0.15]
corr, p_value = spearmanr(human_scores, metric_scores) #A
n_bootstraps = 1000
bootstrap_corrs = []
for _ in range(n_bootstraps): #B
indices = np.random.choice(len(human_scores), len(human_scores), replace=True)
resampled_human = [human_scores[i] for i in indices]
resampled_metric = [metric_scores[i] for i in indices]
boot_corr, _ = spearmanr(resampled_human, resampled_metric)
bootstrap_corrs.append(boot_corr)
lower = np.percentile(bootstrap_corrs, 2.5) #C
upper = np.percentile(bootstrap_corrs, 97.5)
print(f"Spearman's ρ: {corr:.3f}, p-value: {p_value:.3f}")
print(f"95% CI: [{lower:.3f}, {upper:.3f}]")
Use the runnable NumPy and SciPy listing to compute Spearman’s ρ, bootstrap resamples, and percentile-based confidence bounds. section 4.1.2











