A word at the start of a sentence may be capitalized without changing its meaning. Case folding maps capitalization variants to one form—usually lowercase—so they share a token in an NLP vocabulary. This can shrink the vocabulary and let a pipeline combine evidence from different spellings of the same word. section 2.3.2
>>> tokens = ['House', 'Visitor', 'Center'] >>> normalized_tokens = [x.lower() for x in tokens] >>> print(normalized_tokens) ['house', 'visitor', 'center']
Lowercasing each token turns `House`, `Visitor`, and `Center` into `house`, `visitor`, and `center`. Apply the same transformation wherever tokens are compared. section 2.3.2
For document vectors, fold case before counting tokens and building the vocabulary. In a worked corpus, lowercased tokens are collected across documents, deduplicated with a set, and used as columns in a count DataFrame: 482 token occurrences become 246 unique vocabulary entries. Fewer separate entries can mean smaller vectors and less labeled data needed to cover capitalization variants. section 3.2 section 2.3.2
Decide whether the lost distinction matters before folding:
- For a keyword-based greeting matcher, lowercasing both the stored greetings and the user's statement can prevent capitalization alone from blocking a match. It does not handle every greeting, typo, or punctuation pattern. section 1.5.2
- For named entity recognition, keep or otherwise account for capitalization when it helps identify names and proper nouns. Folding can erase distinctions such as `doctor` versus `Doctor`, or `John` versus `john`. section 2.3.2 section 1.5.2
Case folding is a vocabulary and matching choice, not a guarantee that differently capitalized strings mean the same thing. Use it when the simpler representation is worth the information it removes. section 2.3.2
