A token is a smaller text component produced when a document is parsed for full-text search. Tokens can be words, numbers, phrases, or parser-visible symbols; later processing converts relevant tokens into lexemes. section 6.1 section 6.1.1
Inspect tokens with ts_debug
Run SELECT token, description, lexemes, dictionary FROM ts_debug('5 explorers are traveling to a distant galaxy');. Postgres reports 5 as an unsigned integer, words such as explorers as ASCII words, and spaces as space symbols. The same output shows later normalization: explorers becomes explor, traveling becomes travel, and stop words such as are produce no lexeme. section 6.1.1
Listing 6.1 Analyzing tokenization and normalization with ts_debug
SELECT token, description, lexemes, dictionary
FROM ts_debug('5 explorers are traveling to a distant galaxy');
Use the runnable listing to inspect how Postgres splits a sentence and records each token type, lexeme, and dictionary. section 6.1.1
Tokenization fits into a larger search-processing flow:
- Parse the document into tokens, normalize tokens into lexemes, store or index the lexemes, then search them. section 6.1
- Let the configured parser and dictionaries determine token types and normalization behavior. section 6.1.2
