Chapter 7: Number go up (or down)! Correlation and linear regression

Introduction

The chapter introduction defines linear regression as a way to model and predict a linear relationship, explains why it is useful, and introduces correlation as a numerical score of a relationship.

• Linear regression and prediction: Explain that linear regression fits a linear function between independent inputs and a dependent output, then uses the fitted line to predict unseen data when a linear relationship exists.
  Why it matters: This is the basic idea behind the regression methods developed later in the chapter.
• Practical strengths of linear regression: Describe why the introduction presents linear regression as a workhorse: its operations are simple and interpretable, it applies across domains, underpins more advanced models, and can scale to more data and variables.
  Why it matters: These strengths explain why a simple-looking model is worth learning.
• Correlation as a relationship score: State that correlation is closely related to linear regression and represents the relationship between two variables with one calculated number.
  Why it matters: This distinguishes the relationship score introduced here from regression’s fitted line for prediction.

Key terms: linear regression, independent (input) variables, dependent (output) variables, linear relationship, linear operations, scalable, correlation

7.1 Correlation

This section introduces correlation and uses a sports-drink sales scatterplot to show how an upward linear pattern can indicate a relationship without proving causation.

• Correlation directions and absence: Explain what correlation measures and distinguish positive correlation, negative correlation, and no correlation.
  Why it matters: Direction and absence of a relationship mean different things when you interpret data.
• Sports drink scatterplot pattern: Describe what the temperature-versus-sales scatterplot shows and how the line helps make its pattern visible.
  Why it matters: A scatterplot lets you describe a relationship while recognizing that individual observations can vary.
• Correlation and causal claims: Explain why an observed correlation is a clue to a possible relationship, not proof that one variable causes the other.
  Why it matters: This caution keeps a useful pattern from being mistaken for an established cause.
• Relationships beyond straight lines: Distinguish a nonlinear relationship from the straight-line pattern used in the sports-drink example.
  Why it matters: A lack of a straight-line pattern does not necessarily mean there is no relationship.

Key terms: correlation, positive correlation, negative correlation, no correlation, scatterplot, linear pattern, positive linear relationship, noise, causation, coincidental

7.2 The Pearson correlation

This section introduces Pearson’s correlation coefficient, shows how to interpret its value, and cautions that a strong observed correlation is not automatically statistically significant.

• Pearson correlation coefficient: Explain that Pearson’s correlation coefficient, r, measures how strongly two variables follow a linear pattern, and describe its calculation as covariance divided by the product of their standard deviations.
  Why it matters: It gives a numerical way to describe the linear pattern you see in a scatterplot.
• Correlation direction and strength: Describe how values from -1 to 1 indicate the direction and strength of a linear correlation, including what values near zero mean and how stronger correlations appear in scatterplots.
  Why it matters: It lets you interpret both the direction and the apparent strength of a reported correlation.
• Limits of a high correlation: Explain why the sports drink sales and temperature correlation of .9889 indicates a very strong positive linear relationship but should not be accepted uncritically.
  Why it matters: A striking coefficient is not, by itself, enough to justify confidence in a finding.

Key terms: Pearson correlation, correlation coefficient (r), covariance, standard deviations, negative correlation, positive correlation, linear correlation, sports drink sales, temperature, statistically significant

7.3 Hypothesis testing the correlation coefficient

This section explains how to test whether a Pearson correlation is statistically significant and why the p-value depends on both the correlation and the amount of data.

• Correlation hypotheses and p-values: Explain the null and alternative hypotheses for a Pearson correlation test, what its p-value means, and how the sports-drink example is judged against an alpha threshold.
  Why it matters: It lets you interpret a reported correlation test without mistaking a low p-value for proof of causation or for the probability that the null is true.
• Two-tailed correlation test: Describe how the section proposes calculating the correlation p-value manually and why the test uses n − 2 degrees of freedom.
  Why it matters: It connects the reported Pearson p-value to the two-tailed testing ideas used earlier in the book.
• Sample size and correlation evidence: Explain how sample size and how closely points follow a line affect a Pearson correlation p-value, including the two-point example.
  Why it matters: A striking correlation coefficient alone can be misleading when there are very few observations.

Key terms: null hypothesis, alternative hypothesis, Pearson correlation, p-value, alpha threshold, two-tailed t-test, t-distribution, degrees of freedom, n - 2, sample size

7.4 Assumptions of Pearson correlation

This section explains when Pearson correlation is useful, why plots can reveal misleading correlation values, and what to consider when its assumptions fail.

• Pearson correlation assumptions: State the basic conditions for using Pearson correlation: a linear relationship between continuous variables, normally distributed variables, and data points that do not influence one another.
  Why it matters: A correlation number is less useful if the data do not meet the conditions behind it.
• Visual checks for violations: Explain how changing variance, severe outliers, and nonlinear shapes can make a Pearson correlation misleading, using the funnel-shaped example and Anscombe’s quartet.
  Why it matters: Looking at the data can expose problems that a single coefficient hides.
• Responses to assumption violations: Describe the section’s suggested responses to outliers, nonlinear shapes, and changing variance: consider ethical outlier removal, logarithmic transformations, or a nonparametric method.
  Why it matters: A failed assumption calls for a considered response, not blind use of the original number.
• Alternative correlation measures: Distinguish the uses and limitations the sidebar gives for Spearman correlation, Kendall’s correlation, and cosine similarity.
  Why it matters: The choice of measure depends on the data and on what the measure gives up.

Key terms: Pearson correlation, linear, continuous variables, normal distribution, homoscedasticity, heteroscedastic, outliers, Anscombe’s quartet, data cleaning, logarithmic transformations

7.5 Correlation is not causation

This section explains why correlation alone cannot establish causation, how to strengthen a causal claim, and why a correlation may still be useful for prediction.

• Correlation versus causation: Explain why a strong correlation, even with a low p-value, does not show that either variable causes the other. Describe how an unobserved third variable or coincidence could account for the relationship.
  Why it matters: It keeps a striking statistical result from becoming an unsupported causal claim.
• Evidence for causal claims: Explain why a computer's detection of a correlation cannot settle causation, and describe how experiments, new data, and further research can strengthen a causal claim without necessarily proving it.
  Why it matters: It points you toward better evidence while preserving appropriate skepticism about causal claims.
• Correlation for prediction: State that a potentially useful correlation may motivate building a linear regression to make predictions, even though the correlation has not established causation.
  Why it matters: It separates the practical use of a relationship for prediction from a claim about what causes it.

Key terms: correlation, causation, p-value, spurious correlations, causality, randomized double-blind testing, healthy skepticism, linear regression, prediction

7.6 Linear regression

Defines regression as fitting relationships for prediction and linear regression as predicting an output by scaling and adding inputs.

• Linear regression and prediction: Explain that regression fits a functional relationship between one or more independent variables and a dependent variable to make predictions, and that linear regression predicts the dependent variable by scaling and adding the independent variables.
  Why it matters: This distinction gives you the basic idea behind the simple linear regression explained next.

Key terms: regression, linear regression, independent variables, dependent variable

7.7 Simple linear regression

Introduces simple linear regression as a way to predict sports drink sales from temperature, interprets the fitted line, and explains why its predictions still need scrutiny.

• Regression line and prediction: Explain how simple linear regression fits a line to one input variable and one output variable, and distinguish its predictive purpose from measuring correlation.
  Why it matters: This is the shift from describing an association to using it for a prediction.
• Slope and intercept meanings: Describe the roles of m and b in y = mx + b and interpret their fitted values in the temperature and sports drink example.
  Why it matters: Interpreting the coefficients turns a fitted equation into a meaningful—and appropriately qualified—prediction.
• Limits of fitted predictions: Justify why a line that closely fits these data is not automatically a reliable or statistically significant prediction model.
  Why it matters: A convincing-looking line can still give poor predictions when the evidence behind it is inadequate.
• Residual distribution intuition: Describe the section's qualified intuition about normally distributed deviations around an underlying regression line.
  Why it matters: It gives an intuitive meaning to the residuals the section says should be examined.

Key terms: simple linear regression, input variable x, output variable y, correlation, y = mx + b, slope, intercept, tight linear regression, overfitting, generalization

7.8 Interpolation vs. extrapolation

This section distinguishes interpolation from extrapolation and explains why predictions or claims beyond observed data can be unreliable.

• Interpolation and extrapolation boundaries: Distinguish predictions made within the observed data range from predictions made outside it, and explain why the latter are less dependable.
  Why it matters: The distinction helps you tell when a prediction extends beyond the evidence used to make it.
• Sports drink model limits: Explain why the sports drink sales regression should not be treated as a useful model at freezing or extremely hot temperatures.
  Why it matters: A line can produce a numerical answer even where the underlying situation makes that answer unhelpful.
• Extrapolation beyond training data: Describe how the same extrapolation warning applies to more complex models and to claims drawn from their performance on limited tasks.
  Why it matters: The warning applies not just to regression forecasts but also to how we interpret what an AI system can do.

Key terms: interpolation, extrapolation, observed data range, linear regression, y-axis, intercept, range of data, training the model, large language model

7.9 Residuals and sum of squares

Explains how residuals become a sum-of-squares loss and how simple linear regression chooses the line that minimizes it.

• Residuals for a candidate line: Describe how to calculate each residual for a given line and why residuals can have different signs.
  Why it matters: Residuals are the point-by-point errors used to score a regression line.
• Sum of squares loss: Explain how residuals are combined into the sum of squares and what a smaller or larger value says about a line’s fit.
  Why it matters: This loss gives a concrete way to compare how well different lines fit the same data.
• Minimum loss line coefficients: Explain what it means to choose m and b by minimizing the sum of squares, and state the level of calculation this section attributes to simple linear regression.
  Why it matters: It connects the score for a given line to the coefficients a fitting library selects.

Key terms: residuals, ŷ, y-hat, sum of squares, loss function, m and b, minimize, simple linear regression

7.10 Overfitting and bias/variance tradeoff

The section explains why fitting training data perfectly can hurt prediction and how model bias trades off against prediction variability.

• Perfect Fit and Overfitting: Explain why a flexible model that eliminates residuals and reduces the sum of squares to zero may still predict new data poorly.
  Why it matters: A zero training loss does not, by itself, show that a model will be useful for prediction.
• Model Bias Variance Tradeoff: Distinguish model bias from biased data and explain how a model's adherence to a rigid rule trades off against the variability of its predictions.
  Why it matters: The tradeoff explains why allowing a model to follow the data more closely is not always better for prediction.

Key terms: loss function, sum of squares, residuals, noise, biased model, biased data, linear regression, bias-variance tradeoff, variance

7.11 Evaluating a simple linear regression

This section explains how R² and RMSE evaluate a simple linear regression, why graphical checks still matter, and how multiple regression uses beta coefficients.

• R squared and baseline fit: Explain what R² measures and how the section obtains it by squaring the Pearson correlation or comparing the regression’s squared residuals with those of a prediction based on the average of y.
  Why it matters: R² puts the regression’s overall fit in context by comparing prediction using x with a simple average-y baseline.
• Root mean squared error: Describe how RMSE is calculated from a regression’s squared residuals and what a smaller value indicates.
  Why it matters: RMSE gives a way to discuss the typical size of prediction errors rather than only overall explained variation.
• Graphical checks alongside metrics: Justify why evaluating a regression should not stop after calculating R² and RMSE.
  Why it matters: A numerical fit summary is not a substitute for looking at the data.
• Multiple regression beta notation: Describe how multiple linear regression extends the equation to several input variables and identify which beta coefficients are slopes and which is the intercept.
  Why it matters: Beta notation makes a model with several inputs easier to recognize without changing what slopes and an intercept mean.

Key terms: R2, coefficient of determination, Pearson correlation, squared residuals, dependent variable, independent variable, root mean squared error, RMSE, residuals, sum of squares

7.12 A real-world example

The section uses FAA bird-strike data to show why an apparent altitude pattern does not necessarily support Pearson correlation or linear regression.

• Bird Strike Data Limitations: Describe how the FAA database can reveal patterns in reported strikes while limiting conclusions about what predicts a strike.
  Why it matters: The available records affect which questions the analysis can answer.
• Altitude Counts And Binning: Explain why relating HEIGHT to a number-of-strikes variable is awkward in this dataset, and what creating 100-foot bins does and does not solve.
  Why it matters: Making a count variable does not automatically make it appropriate for a linear model.
• Plots Versus Promising Statistics: Trace why neither the initial binned analysis nor the logarithmic transformation yielded a useful linear model, despite low p-values and an improved correlation coefficient.
  Why it matters: A scatterplot can expose problems that an attractive coefficient or p-value conceals.
• Pattern Without Linear Fit: Distinguish the observed altitude pattern from a successful linear relationship and explain why the author stops trying to force this model.
  Why it matters: Rejecting a poor linear fit does not require ignoring a real pattern in the observations.

Key terms: FAA bird strike database, self-reported, flights where no bird strikes occurred, HEIGHT, 100-foot bins, dependent variable, Pearson correlation, linear regression, p-value, logarithmic transformation

7.13 Summary