liveBook · Manning

Standard analyzer

As explained in Elasticsearch in Action, Second Edition

Elasticsearch's built-in text analyzer that applies standard tokenization and related default text processing.

The standard analyzer is Elasticsearch’s default text analyzer. Use it when ordinary text such as titles, reviews, or descriptions should be searchable as individual words without first designing a custom analysis chain. It splits input around whitespace, punctuation, and grammar, then lowercases the resulting tokens. section 7.0 section 7.3

Indexing a snack-and-drink description

Given `Hot cup of ☕ and a 🍿 is a Weird Combo :(!!`, the analyzer produces tokens such as `hot`, `cup`, `of`, `☕`, `and`, `a`, `🍿`, `is`, `weird`, and `combo`. Words are lowercased; the smiley and exclamation marks disappear; the coffee and popcorn emojis remain searchable tokens. Stop words such as `of`, `and`, and `a` remain because stop-word removal is disabled by default. section 7.3.1

Listing 7.4 Testing the standard analyzer with an explicit call
GET _analyze
{
  "analyzer": "standard",      #1
  "text": "Hot cup of ☕ and a 🍿 is a Weird Combo :(!!"  
}

Run this `_analyze` request to inspect standard-analyzer tokens before indexing. Naming the analyzer explicitly makes the intended processing clear; omitting it uses the same default behavior. section 7.2.4 section 7.3.1

Testing matters because indexed terms determine what can match later. For example, `James Bond 007` becomes `james`, `bond`, and `007`. The response also reports token type, position, and character offsets, which helps explain unexpected search behavior before production indexing. section 7.2.4

Use the standard analyzer with these search expectations in mind:

  • A text title such as `The Godfather` is stored as separate lowercase terms, including `the` and `godfather`, rather than as one exact title value. section 9.1.2
  • A term-level query is not analyzed, so `The Godfather` and even `the godfather` do not match those separate indexed tokens; `godfather` can match exactly. section 9.2.2
  • A match query analyzes its search text with the indexing analyzer unless configured otherwise, so `JAVA`, `java`, and mixed-case variants receive the same lowercase treatment. section 10.5.3

The default is a practical baseline, not a complete language strategy. It does not remove stop words by default and does not stem words. When those behaviors are needed, define a named analyzer of type `standard` with an English or Hindi stop-word set, a custom stop-word file, or a chosen maximum token length. For more specialized requirements, Elasticsearch also provides language-specific and other built-in analyzers, plus custom combinations of tokenizers and filters. section 7.3.1 section 7.0 section 7.9

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Explains token output, punctuation and emoji handling, stop-word configuration, and token-length customization, while deferring component anatomy details.
Examples
Demonstrates snack-and-emoji text, `James Bond 007`, and `The Godfather` movie-title searches.
What it assumes
Introduces analyzer behavior from raw text and JSON requests without requiring prior analysis-chain knowledge.
What you can do after
Diagnose/debug with it by inspecting emitted tokens and predicting exact-term versus analyzed-match behavior.
How it connects
Connects text analysis with tokenization, lowercase filtering, stop words, term queries, match queries, and the `_analyze` API.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.