Character-based string similarity compares strings by counting character-level operations—insertions, deletions, and substitutions—needed to transform one string into another. Higher similarity scores indicate closer matches. The approach suits short strings, while longer strings are often compared as tokens or n-grams. section 8.4
This comparison is useful when categories contain typos or many distinct values: similar strings can be grouped or converted into numerical features instead of receiving unrelated categorical codes. section 8.4
A practical workflow is:
- Choose character-level similarity for short categorical strings; consider token-based methods for longer strings. section 8.4
- Encode noisy or high-cardinality categories with dirty_cat, using SimilarityEncoder or related substring-based encoders. section 8.4
- Evaluate the encoded features inside a modeling pipeline rather than assuming similarity improves every model. section 8.4
Employee salary categories
A salary dataset contains 385 employee position titles. A pipeline compares SimilarityEncoder, MinHashEncoder, and GapEncoder alongside low-cardinality and continuous features; reported R² scores are 0.8996, 0.8997, and 0.8895 respectively. section 8.4
