Tokenization splits a document into discrete tokens before full-text search. In PostgreSQL, the sentence “5 explorers are traveling” becomes “5,” “explorers,” “are,” and “traveling.” Normalization then converts tokens into lexemes, while storage and indexing prepare them for queries. section 6.1
PostgreSQL’s parser identifies token types before dictionaries normalize them:
- Numbers such as “5” become numeric tokens and remain {5}. section 6.1.1
- Words such as “explorers” and “traveling” are normalized by English stemming to {explor} and {travel}. section 6.1 section 6.1.1
- Stop words such as “are,” “to,” and “a” produce no lexemes. section 6.1
Inspect tokens with ts_debug
Run SELECT token, description, lexemes, dictionary FROM ts_debug('5 explorers are traveling to a distant galaxy'); to inspect each token, its type, resulting lexeme, and dictionary. PostgreSQL uses the default built-in parser unless another configuration is selected.
