A high-cardinality categorical feature has many distinct labels relative to the dataset size or available memory. There is no universal cutoff: IDs, zip codes, product names, geographical names, and similar columns can become high cardinality when their unique-value count is large enough to make ordinary encoding impractical. section 6.1.3 section 2.1.2
Start by counting unique values in every categorical column, then separate low- and high-cardinality fields. This matters because one-hot encoding creates one binary column per label; with many labels, the result is excessively wide and sparse. Sparse representations can make learning harder, particularly for neural networks and online algorithms, and may exceed memory when converted to dense data. section 4.2.1 section 6.1.3
Choose treatment according to cardinality, model, and operational cost:
- One-hot encode low-cardinality categories when the resulting number of binary features remains manageable. section 4.2.1
- Use ordinal encoding with complex tree-based models when recursive splits can find useful partitions, while recognizing that many categories can make those partitions difficult to learn. section 6.1.3
- Use target encoding, also called mean encoding, as a standard alternative for high-cardinality fields when one-hot or ordinal encoding becomes problematic. section 6.1.3
- Avoid maintaining separate preprocessing pipelines for every model unless evidence shows the performance gain justifies the added maintenance effort. section 7.2.2
Classifying Airbnb location fields
In an Airbnb dataset with 48,895 rows, neighbourhood has 221 unique values, compared with 5 for neighbourhood_group and 3 for room_type. Treat neighbourhood as high cardinality while one-hot encoding the smaller fields. A derived coordinates category can be even larger: binning latitude and longitude produces 2,259 distinct coordinate values. In a tree-based pipeline, room_type is one-hot encoded, mid-sized categorical fields are ordinal encoded, and coordinates are target encoded. section 4.2.1 section 6.1.3 section 7.2.2
Listing 6.3 Creating a high cardinality categorical feature
def bin_2_cat(feature, bins=100):
min_value = feature.min()
bin_size = (feature.max() - min_value) / bins
bin_values = (feature - min_value) / bin_size
return bin_values.astype(int) #1
data['coordinates'] = (
bin_2_cat(data['latitude']) * 1000
+ bin_2_cat(data['longitude']
) #2
high_card_categorical += ['coordinates']
print(data[high_card_categorical].nunique()) #3
This listing constructs a high-cardinality coordinates category by binning latitude and longitude, then counts its distinct values alongside neighbourhood. section 6.1.3
A practical rule of thumb is to question a feature once its classes exceed roughly 5%–10% of dataset rows; a threshold such as 512 may suit some datasets but be too high for smaller ones. The useful threshold depends on both row count and the encoded feature size that available memory can handle. section 6.1.3
