A dimension is an independent feature or coordinate describing each observation. In a dataset, many dimensions make data harder to manage and increase clustering costs. section 2.7
Practical ways to manage dimensions include:
- Select significant variables with supervised regression or a decision tree. section 2.7
- Transform dimensions with principal component analysis or singular value decomposition. section 2.7
- Combine logically related variables into derived measures, or remove variables through manual selection and correlation analysis. section 3.4
- Encode categorical dimensions numerically with one-hot encoding, while checking whether many categories create excessive sparse columns. section 2.7
Retail customer dimensions
A customer record with Sales, Invoices, and Number of Items has three dimensions. Replace two of them with Average Transaction Value (ATV = Sales / Invoices) and Average Basket Size (ABS = Number of Items / Invoices), reducing the representation to two dimensions. For customer 1, Sales = 100 and Invoices = 2 produce ATV = 50; five items produce ABS = 2.5. section 3.4
Figure 2.36 Using one-hot encoding to convert categorical variables into numeric ones
Use the one-hot encoding figure to inspect how one categorical dimension becomes multiple binary dimensions. section 2.7
Figure 3.3 In the first table, we have the sales, invoices, and number of items as the variables. In the second table, they have been combined to create new variables.
Use the retail figure to compare the original variables with their reduced, derived representation. section 3.4
