Correlation describes a mutual relationship between two variables: changes in one tend to correspond with changes in the other. Highly correlated variables often provide similar information, making one a candidate for removal during dimensionality reduction. section 3.4.2
Use correlation as a filtering signal, not an automatic deletion rule:
- A retail store may find daily invoices highly correlated with daily sales, so either variable could be considered for removal. section 3.4.2
- Students who study more hours generally receive better grades, illustrating a positive association. section 3.4.2
- Check business meaning and consult stakeholders before dropping a variable, because correlation alone is insufficient. section 3.4.2
Reducing redundant customer variables
Customer data containing sales, invoices, and item counts can be reshaped into measures such as average transaction value and average basket size. Correlation helps identify variables supplying overlapping information, while business judgment determines whether replacement or removal is appropriate. section 3.4 section 3.4.2
Figure 3.3 In the first table, we have the sales, invoices, and number of items as the variables. In the second table, they have been combined to create new variables.
Inspect the before-and-after customer tables to see how several related variables can be combined during manual dimensionality reduction. section 3.4
