Standard tokenizer

A standard tokenizer is a built-in Elasticsearch analysis component that turns text into separate tokens using predefined word-boundary rules. It splits on whitespace and punctuation such as commas, hyphens, colons, and semicolons, making text searchable as individual terms rather than one unbroken string. section 7.7.1 section 3.1.2

Tokenization supports matching a query against words stored from a text field. For example, a synopsis containing “Peter Piper picked a peck of Pickled-peppers!” can be represented as tokens including peter, piper, picked, pickled, and peppers, so searches containing Peter and Peppers can match it. section 3.1.2

POST _analyze
{
  "text": "Hello,cruel world!",
  "tokenizer": "standard"
}

Run this request to inspect how standard tokenization separates punctuation-delimited and whitespace-delimited words. section 7.7.1

The standard tokenizer is one component, not the whole standard analyzer. The default standard analyzer also applies a lowercase token filter: “Weird Combo” becomes lowercase tokens, while the tokenizer removes the smiley and exclamation marks. Emoji in the example remain indexed as tokens. Stop words are not removed by default. section 7.3.1

Use the distinction when configuring analysis:

• Choose the standard tokenizer when word-boundary splitting on whitespace and punctuation fits the field. section 7.7.1
• Use the standard analyzer when its built-in tokenization and lowercasing behavior are appropriate; it is Elasticsearch’s default analyzer. section 7.3.1
• Add a stop-word configuration when common words should be omitted; stop filtering is disabled by default. section 7.3.1
• Create a custom tokenizer when tokens must be split at a chosen maximum length. section 7.7.1

PUT index_with_custom_standard_tokenizer
{
  "settings": {
    "analysis": {
      "analyzer": {
        "custom_token_length_analyzer": {                #1
          "tokenizer": "custom_token_length_tokenizer"
        }
      },
      "tokenizer": {
        "custom_token_length_tokenizer": {
          "type": "standard", 
          "max_token_length": 2                          #2
        }
      }
    }
  }
}

This index settings example names a standard tokenizer and sets max_token_length to 2 for a custom analyzer. section 7.7.1

Check analysis output before indexing whenever token boundaries affect search behavior. The _analyze API can test the implicit default analyzer or an explicitly named analyzer, helping verify punctuation handling, lowercasing, retained emoji, and configured stop words or token length. section 7.3.1 section 7.7.1