A full-text search configuration maps token types to dictionaries that normalize tokens into lexemes. PostgreSQL provides configurations for many human languages, including English and Russian. Functions such as ts_debug, to_tsvector, to_tsquery, and ts_headline accept an optional regconfig argument; without one, they use default_text_search_config. section 6.1.2 section 6.3.2
Use configurations to inspect or control normalization behavior:
- Check the active configuration with SHOW default_text_search_config;. section 6.1.2
- Pass a configuration explicitly when tokenizing or querying, such as ts_debug('english', document) or to_tsquery('russian', query). section 6.1.2 section 6.3.2
- Inspect dictionary assignments with \dF+ configuration in psql; English uses simple for several non-word token types and english_stem for word types. section 6.1.2
- Choose the same configuration when generating stored tsvector lexemes so later searches use compatible normalization. section 6.2.2
Compare English normalization
SELECT token, lexemes, dictionary FROM ts_debug('english', '5 explorers are traveling to a distant galaxy'); The English configuration produces lexemes such as explor, travel, distant, and galaxi through english_stem, while the number 5 uses simple. section 6.1.2
Text search configuration "pg_catalog.english"
Parser: "pg_catalog.default"
Token | Dictionaries
-----------------+--------------
asciihword | english_stem
asciiword | english_stem
email | simple
file | simple
float | simple
host | simple
hword | english_stem
hword_asciipart | english_stem
hword_numpart | simple
hword_part | english_stem
int | simple
numhword | simple
numword | simple
sfloat | simple
uint | simple
url | simple
url_path | simple
version | simple
word | english_stem
Inspect the complete English mapping from token types to dictionaries. section 6.1.2
token | description | lexemes | dictionary
----------------+-------------------+----------------+--------------
5 | Unsigned integer | {5} | simple
| Space symbols | |
исследователей | Word, all letters | {исследовател} | russian_stem
| Space symbols | |
путешествуют | Word, all letters | {путешеств} | russian_stem
| Space symbols | |
к | Word, all letters | {} | russian_stem
| Space symbols | |
далёкой | Word, all letters | {далек} | russian_stem
| Space symbols | |
галактике | Word, all letters | {галактик} | russian_stem
. | Space symbols | |
(12 rows)
Compare Russian token normalization and stemming with the English example. section 6.1.2
