A tokenizer divides text into discrete tokens according to rules such as whitespace, punctuation, letter boundaries, patterns, or custom separators. In Elasticsearch, those tokens are usually the individual words that can later support full-text search. section 7.2.1 section 7.7
Tokenization is one phase of text analysis. Character filters can first clean raw text; the tokenizer then emits tokens; token filters can modify, enrich, or normalize those tokens before they are stored in an inverted index. Search queries undergo the same analysis path. section 7.2.3 section 7.9
Follow the analyzer pipeline to see where a tokenizer turns cleaned text into tokens before token filters process them. section 7.2.3
Choose tokenization rules to fit the shape of the input:
- Use whitespace or punctuation boundaries to separate ordinary sentence words. section 7.2.1
- Use custom delimiters for cases such as hyphens, non-letters, or colons; for example, pickled-peppers can become pickled and peppers. section 7.2.1
- Use specialized tokenizers when needed: pattern splits on regex matches, uax_url_email preserves URLs and emails, keyword leaves text intact, and path_hierarchy handles path separators. section 7.7.3
Test a custom analyzer
A custom analyzer can strip HTML, tokenize with the standard tokenizer, and uppercase the resulting tokens. Testing it with <H1>HELLO, WoRLD</H1> produces ["HELLO", "WORLD"]. Define the analyzer when creating the index, then send text to its _analyze endpoint to inspect the result. section 7.4 section 4.5.1
Every analyzer has exactly one tokenizer, but it may have zero or more character filters and token filters. Elasticsearch uses a standard analyzer by default when no analyzer is specified; custom analyzers mix a chosen tokenizer with available filters. section 7.4 section 7.9
Tokenization alone makes text searchable as terms, but it does not by itself address synonyms, plurals, stemming, or similar language concerns. Those are handled during normalization through token filters and related analysis choices. section 7.2.1 section 4.5.1
