When a user searches a synopsis for “Peter” or “pickled peppers,” Elasticsearch must first turn the text into searchable pieces. Tokenization splits a text field into individual tokens according to rules such as whitespace, punctuation, letters, delimiters, or patterns. section 3.1.2 section 7.2.1
A tokenizer performs this chopping step. A whitespace tokenizer separates words at whitespace, while other tokenizers can split on nonletters, colons, digits, or custom separators. Tokenization creates tokens but does not perform later enhancements such as synonym handling, stemming, or other normalization. section 7.2.1 section 3.1.2
From synopsis text to tokens
The input “Peter Piper picked a peck of Pickled-peppers!” becomes [peter, piper, picked, a, peck, of, pickled, peppers]. The displayed result includes lowercasing and punctuation removal, so it illustrates tokenization together with analysis steps beyond merely splitting the character stream. section 3.1.2
Choose tokenization rules according to the searchable text:
- Use whitespace-style splitting when word boundaries are primarily whitespace. section 3.1.2 section 7.2.1
- Use separators or patterns when terms contain hyphens, punctuation, colons, digits, or other meaningful boundaries; for example, “pickled-peppers” can become “pickled” and “peppers,” while “K8s” can become “K” and “s.” section 7.2.1
- Remember that tokenization alone cannot answer synonym, plural, or related-word queries; normalization continues the analysis. section 7.2.1
