The Hughes phenomenon describes a classifier whose performance improves as variables are added, reaches an optimum, and then declines when more variables are included with the same training data. It shows why adding every available attribute can reduce accuracy rather than improve it. section 3.2
Additional dimensions can make a supervised learning problem harder by increasing:
- Mathematical complexity, including more complicated calculations between data points. section 3.2
- Sparsity, because larger spaces contain fewer populated or complete observations. section 3.2
- The risk that insignificant variables add complexity without contributing to accuracy. section 3.11 section 3.2
Adding attributes to a dataset
A dataset might begin with gender, then add age, education, and address. Its dimensionality increases with each attribute, but classifier performance eventually peaks and then falls if the added variables do not justify their cost. section 3.2
