liveBook · Manning

Max token length

As explained in Elasticsearch in Action, Second Edition

A tokenizer configuration setting that specifies the maximum number of characters permitted in a token before it is split.

Max token length limits the number of characters in a token. The standard tokenizer defaults to 255 characters; longer tokens are split when a smaller limit is configured. section 7.7.1

Split a long token

Configure a custom standard tokenizer with "max_token_length": 2. Analyzed text such as "Bond" becomes "Bo" and "nd", preserving the requested two-character limit. section 7.7.1

POST weird_combos/_doc 
{
  "text": "Hot cup of ☕ and a 🍿 is a Weird Combo :(!!"
}

Inspect the complete index settings for a custom analyzer and standard tokenizer with a two-character maximum. section 7.7.1

["hot", "cup", "of", "☕", "and", "a", """🍿""", "is", "a", "weird", "combo"]

Run the analyzer against "Bond" and verify that the configured token-size limit produces two tokens. section 7.7.1

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Defines the 255-character default and demonstrates splitting a four-character token into two-character pieces, without broader tokenizer internals.
Examples
Uses a custom tokenizer that turns "Bond" into "Bo" and "nd" with a two-character limit.
What you can do after
Apply it via Elasticsearch index settings and verify tokenization with the _analyze API.
How it connects
Connects standard analyzers, standard tokenizers, custom analyzers, index settings, and the _analyze API.
Cover of Elasticsearch in Action, Second Edition
liveBookmanning.com
Read Elasticsearch in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.