A full-text search dictionary converts parsed tokens into lexemes or maps them to an empty lexeme when no searchable term applies. Dictionaries can preserve tokens unchanged, stem words, or remove stop words while retaining their positions. section 6.1.1
PostgreSQL selects dictionaries through a full-text search configuration:
- The simple dictionary leaves values such as 5 unchanged. section 6.1.1
- The english_stem dictionary stems explorers to {explor} and traveling to {travel}. section 6.1.1
- Stop words such as are, to, and a become {} while preserving their original positions. section 6.1.1
- Configurations choose dictionaries by token type; the english configuration uses simple for numbers and english_stem for ordinary words. section 6.1.2
Inspect dictionary decisions
Run SELECT token, description, lexemes, dictionary FROM ts_debug('english', '5 explorers are traveling to a distant galaxy');. The output shows simple producing {5}, english_stem producing {explor}, {travel}, and {galaxi}, and stop words producing {}. section 6.1.2
Listing 6.1 Analyzing tokenization and normalization with ts_debug
SELECT token, description, lexemes, dictionary
FROM ts_debug('5 explorers are traveling to a distant galaxy');
Inspect each token, its type, resulting lexemes, and the dictionary selected during tokenization and normalization. section 6.1.1
