dirty_cat is an open-source Python library for encoding noisy or high-cardinality categorical strings. It is useful when categories contain typos, inconsistent spellings, or too many distinct values for straightforward one-hot encoding. section 8.0 section 8.4
The central idea is to compare strings rather than treat every spelling as unrelated. Character-based similarity counts insertions, deletions, and substitutions, while token-based methods compare substrings or words such as n-grams. Similarity can group related categories and reduce the effect of typos. section 8.4
dirty_cat provides three specialized encoders:
- SimilarityEncoder creates a one-hot-like representation based on string similarities. section 8.4
- GapEncoder represents categories through frequently co-occurring substring combinations. section 8.4
- MinHashEncoder applies hashing techniques to substrings and supports a chosen number of components. section 8.4
Employee salary prediction
A salary-regression data set contains 9,211 rows and six features. The employee_position_title column has 385 categories, making it a high-cardinality feature. SimilarityEncoder, MinHashEncoder, and GapEncoder can be compared in an ensemble with low-cardinality encoders, a scaler, and XGBRegressor. Reported R² scores are 0.8996, 0.8997, and 0.8895 respectively. section 8.4
