A lexeme is a normalized linguistic unit that PostgreSQL derives from text for full-text representation and matching. Normalization can remove case distinctions, stem word variations, and discard stop words. section 6.1 section 6.1.1
PostgreSQL produces lexemes through this search-preparation flow:
- Tokenizes a document into words, numbers, phrases, and other smaller components. section 6.1
- Normalizes tokens with dictionaries, applying stemming and stop-word removal according to a text-search configuration. section 6.1.1 section 6.1.2
- Stores the resulting lexemes in a tsvector, including original word positions that support phrase and proximity searches. section 6.2.1
- Matches stored lexemes against normalized tsquery values with the @@ operator. section 6.3
Inspecting normalized words
SELECT token, description, lexemes, dictionary FROM ts_debug('5 explorers are traveling to a distant galaxy'); The output includes {5}, {explor}, {travel}, {distant}, and {galaxi}; words such as “are,” “to,” and “a” produce no lexemes. The english_stem dictionary changes “explorers” to “explor” and “traveling” to “travel.” section 6.1.1
Generating and storing movie lexemes
SELECT to_tsvector('The explorers must save the fragile peace between Earth and the aliens.'); PostgreSQL returns lexemes such as 'explor', 'fragil', 'peac', and 'alien', while preserving positions. For reusable movie searches, store the generated value in a tsvector column, optionally as a stored generated column combining the name and description, then index it with GIN or GiST. section 6.2.1 section 6.2.2 section 6.6.1
Listing 6.1 Analyzing tokenization and normalization with ts_debug
SELECT token, description, lexemes, dictionary
FROM ts_debug('5 explorers are traveling to a distant galaxy');
Run the ts_debug listing to inspect each token, its type, generated lexemes, and selected dictionary. section 6.1.1
