Correlation measures how two variables vary together. Unlike covariance, it puts that association on a standardized scale. A correlation coefficient ranges from –1 to 1: its sign gives the direction of the relationship, while values farther from 0 indicate a stronger association. Correlation does not, by itself, establish that one variable causes the other. section 3.2.2 section 3.4
Use correlations when a hypothesis asks whether variables are associated—for example, whether daily service reports rise with temperature. They can reveal patterns worth investigating before predictive modeling. Unlike descriptive summaries, they quantify a relationship; unlike a controlled experiment, they do not isolate a cause. section 3.2 section 3.4 section 4.3
Choose a coefficient that fits the variables and the shape of their relationship:
- Pearson’s r measures a linear relationship between two continuous variables. Check the relationship with a scatterplot: a nonlinear pattern can produce a misleadingly weak Pearson coefficient. section 4.3.1
- Spearman’s correlation compares ranks and is often used for ordinal or nonlinear relationships. Kendall’s tau also measures ordinal association through ranked pairs. section 4.3.1
- Point-biserial correlation addresses one binary and one continuous variable. section 4.3.1
Daily rat sightings and weather
Join New York City daily rat-sighting counts to daily weather observations, then inspect a Pearson correlation matrix and scatterplots. Sightings have a strong positive linear association with high and low temperatures, a weaker negative association with wind speed, and little association with humidity or precipitation. High and low temperatures are themselves correlated at nearly r = 1, so highlighting both would add little to a simple account of the pattern. None of these results shows that weather causes sightings. section 3.2.2 section 9.2
weather = pd.read_csv("weather.csv", index_col=0) #1
weather = weather.reset_index()
weather["day"] = pd.to_datetime(weather["day"], format="%m/%d/%y")
rats_weather = pd.merge(
weather, rats, how="left", on="day"
).fillna(0) #2
corrs = rats_weather.corr() #3
corrs.style.background_gradient(cmap="RdBu", vmin=-1)
The pandas workflow joins observations by day and displays a color-coded correlation matrix. Inspect individual relationships with scatterplots before treating matrix values as a complete explanation. section 3.2.2 section 4.3.1
For stakeholders, pair the coefficient with a visualization and a plain-language statement of what it does—and does not—support. Watch for causal claims or interpretations driven by prior beliefs. Strong associations can suggest predictors for a regression model, but a predictive relationship is still not automatically a causal one. section 4.3.3 section 9.2
