Lexicon normalization

As explained in Data Without Labels

Lexicon normalization maps variant word forms to a common root or canonical form while preserving their shared lexical meaning.

Lexicon normalization reduces variant word forms to a shared root or canonical form when they preserve the same core meaning. It helps clean text data while keeping business context in view. section 7.6

Two common approaches support normalization:

• Stemming applies suffix-removal rules such as removing “es,” “ing,” “ly,” or “ed.” It is fast and simple, but can produce inaccurate spellings, such as mapping “studies” to “studi.” section 7.6
• Lemmatization maps words to their dictionary or canonical form. It maps “eats,” “eating,” and “eaten” to “eat,” generally improving results at the cost of more time. section 7.6

Figure 7.3 “Ate,” “eaten,” “eats,” and “eating” all have the same root word: “eat.” Stemming and lemmatization can be used to get the root word.

Compare “ate,” “eaten,” “eats,” and “eating” as variants that can map to the root word “eat.” section 7.6

How Data Without Labels covers it

How deep it goes

Defines normalization and contrasts stemming rules with slower, more accurate lemmatization, but does not provide implementation code.

Examples

Uses “ate,” “eaten,” “eats,” and “eating” mapping to “eat,” plus “studies” becoming “studi” through stemming.

What it assumes

Assumes familiarity with words and text data while emphasizing business context during cleaning decisions.

What you can do after

Apply it via stemming or lemmatization to consolidate word forms during text-data preparation.

How it connects

Connects text cleaning with stemming, lemmatization, business context, and data encoding.

liveBookmanning.com

Read Data Without Labels on liveBook Get the book at manning.com

Related concepts

• lemmatizationhas part · Lemmatization is the linguistic normalization of inflected word forms to their canonical dictionary forms, called lemmas.
• data cleaningpart of · Data cleaning is the process of detecting and correcting, removing, transforming, or otherwise handling inaccurate, incomplete, inconsistent, duplicate, or unsuitable data.
• stemminghas part · Stemming is a text-normalization technique that heuristically reduces inflected or derived word forms to a shared stem, which may not be a valid word.

This explainer was generated by Manning's knowledge graph from the text of Data Without Labels. The section links point to the parts of the book it draws on.