liveBook · Manning

Token filter

As explained in Elasticsearch in Action, Second Edition

A token filter is a text-analysis component that transforms, removes, or adds tokens after tokenization.

A token filter transforms the tokens emitted by a tokenizer. It runs after character filtering and tokenization, before finalized tokens are placed in an inverted index. The same analysis pipeline can run for indexed text and search queries, so token filters shape how full-text terms are normalized and matched. section 7.2.3 section 7.9

Use token filters to lowercase or uppercase terms, add synonyms, stem words, or remove apostrophes and punctuation. An analyzer has exactly one tokenizer but may have zero or more token filters, allowing a chain tailored to a field’s search requirements. section 7.8 section 7.9

Test a filter chain

Attach filters directly to a tokenizer with the _analyze API. With a standard tokenizer and the uppercase then reverse filters, bond becomes DNOB: uppercase produces BOND, and reverse produces DNOB. section 7.8

Listing 7.47 Adding a token filter with the tokenizer
GET _analyze
{
  "tokenizer" : "standard",
  "filter" : ["uppercase","reverse"],
  "text" : "bond"
}

This request shows the smallest useful experiment: choose a tokenizer, list filters in execution order, and inspect the resulting tokens. section 7.8

For reusable field analysis, declare filters in a custom analyzer’s filter array when creating an index. A custom analyzer can combine optional character filters, one tokenizer, and token filters; for example, stripping HTML, splitting text with the standard tokenizer, then uppercasing HELLO and WoRLD yields HELLO and WORLD. section 7.4

Token filters are particularly useful for full-text fields. A standard analyzer lowercases terms by default, while an English-oriented analyzer can stem related forms into roots such as movi, hilari, and witti. Test analysis before committing a mapping, especially when language-specific stemming is involved. section 4.5.1 section 7.9

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Explains filter placement, transformations, chaining, custom-analyzer configuration, and _analyze testing; defers most available filters to documentation.
Examples
Demonstrates bond becoming DNOB and HTML-wrapped HELLO, WoRLD becoming HELLO and WORLD.
What it assumes
Introduces token filters through plain analyzer-pipeline steps without requiring prior Elasticsearch knowledge.
What you can do after
Apply it via Elasticsearch custom analyzers and the _analyze API to test token transformations.
How it connects
Connects token filters with analyzers, character filters, tokenizers, inverted indexes, stemming, synonyms, and the _analyze API.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.