A low-cardinality categorical feature contains relatively few distinct categories. In tabular preprocessing, identifying it helps determine whether one-hot encoding is practical: each category becomes a binary column without creating excessive dimensionality or sparsity. section 4.2.1 section 2.1.2
Use the number of unique values to classify categorical columns before modeling:
- Count distinct values with pandas using data[categorical].nunique(). section 4.2.1
- Treat features with relatively few categories as low cardinality; the Airbnb example identifies neighbourhood_group with 5 values and room_type with 3. section 4.2.1
- Separate features with many categories, such as neighbourhood with 221 values, instead of applying the same encoding automatically. section 4.2.1
- Apply one-hot encoding to the low-cardinality group; features with more than 20 unique values may create excessive columns and sparse data. section 4.2.1 section 6.1
Airbnb NYC feature split
A practical split is low_card_categorical = ['neighbourhood_group', 'room_type'] and high_card_categorical = ['neighbourhood']. A preprocessing pipeline can one-hot encode the first list while leaving the high-cardinality feature apart for a different treatment. section 4.2.1 section 6.1
Listing 6.1 Reprising the Airbnb NYC dataset
import numpy as np
import pandas as pd
from sklearn.preprocessing import (
StandardScaler,
OneHotEncoder,
OrdinalEncoder
)
from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
data = pd.read_csv("./AB_NYC_2019.csv")
excluding_list = ['price', 'id', 'latitude', 'longitude', 'host_id',
'last_review', 'name', 'host_name'] #1
low_card_categorical = [
'neighbourhood_group',
'room_type'
] #2
high_card_categorical = ['neighbourhood'] #3
continuous = [
'minimum_nights',
'number_of_reviews',
'reviews_per_month',
'calculated_host_listings_count',
'availability_365'
] #4
target_mean = (
(data["price"] > data["price"].mean())
.astype(int)
) #5
target_median = (
(data["price"] > data["price"].median())
.astype(int)
) #6
target_multiclass = pd.qcut(
data["price"], q=5, labels=False
) #7
target_regression = data["price"] #8
categorical_onehot_encoding = OneHotEncoder(handle_unknown='ignore')
categorical_ord_encoding =
OrdinalEncoder(handle_unknown="use_encoded_value", unknown_value=np.nan)
numeric_standardization = Pipeline([('StandardScaler', StandardScaler()),
('Imputer',
SimpleImputer(strategy="constant", fill_value=0))])
column_transform = ColumnTransformer(
[
('low_card_categories',
categorical_onehot_encoding,
low_card_categorical),
('high_card_categories',
categorical_ord_encoding,
high_card_categorical),
('numeric',
numeric_standardization,
continuous)
],
remainder='drop',
verbose_feature_names_out=True,
sparse_threshold=0.0) #9
lm_column_transform = ColumnTransformer(
[
('low_card_categories',
categorical_onehot_encoding,
low_card_categorical),
('numeric',
numeric_standardization,
continuous)
],
remainder='drop',
verbose_feature_names_out=True,
sparse_threshold=0.0) #10
Inspect the complete pandas and scikit-learn setup, including low- and high-cardinality feature lists and preprocessing components. section 6.1
