Stop-word removal

Q: How deeply does Elasticsearch in Action, Second Edition cover stop-word removal?

Explains default behavior, index configuration, language presets, custom files, and token verification, without descending into filter internals.

As explained in Elasticsearch in Action, Second Edition

A token filter that removes configured stop words from analyzed text.

Stop-word removal filters configured words from analyzed text. In Elasticsearch, the standard analyzer includes a stop token filter, but disables it by default; enabling it removes common words such as “of,” “a,” and “is.” section 7.3.1

Enable English stop words

Configure the analyzer during index creation with "stopwords": "_english_", then call the index’s _analyze endpoint to verify that configured words disappear from the token output. section 7.3.1

Listing 7.5 The `standard` analyzer with a stop words filter enabled

PUT my_index_with_stopwords
{
  "settings": {
    "analysis": {
      "analyzer": {                    #1
        "standard_with_stopwords":{    #2
          "type":"standard",           #3
          "stopwords":"_english_"      #4
        }
      }
    }
  }
}
#1 Sets the analyzer for the index
#2 Names the analyzer
#3 The standard analyzer type
#4 Enables the English stop words filter

Use the analyzer definition to see how stop-word removal is enabled alongside the standard tokenizer and other analysis settings. section 7.3.1

Choose the stop-word source to match the search domain:

Use language presets such as English or Hindi when the language’s standard stop words fit the application. section 7.3.1
Use stopwords_path to load a custom list, such as domain-specific or blacklisted terms, from a file. section 7.3.1
Test the resulting tokens with _analyze before relying on the analyzer for indexed text. section 7.3.1

How Elasticsearch in Action, Second Edition covers it

Cover of Elasticsearch in Action, Second Edition

liveBook manning.com

Read Elasticsearch in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionA stop word filter is a text-processing filter that removes selected high-frequency function words from tokenized text.
liveBook manning.com

Starting Data Analytics with Generative AI and PythonStopword removal is the text-preprocessing technique of excluding selected common function words from analyzed text.
liveBook manning.com

Data Without LabelsStop word removal is the elimination of very common words that carry relatively little task-relevant meaning from text.

Compare how 4 books explain stop-word removal →

Related concepts

standard analyzerpart of · The standard analyzer is a Lucene text analyzer that tokenizes text at word boundaries, normalizes terms, and removes punctuation.
data cleaningpart of · Data cleaning is the process of detecting and correcting, removing, transforming, or otherwise handling inaccurate, incomplete, inconsistent, duplicate, or unsuitable data.
NLTKuses · NLTK is an open-source Python library providing tools, algorithms, corpora, and lexical resources for natural-language processing and text analysis.
latent Dirichlet allocationuses · Latent Dirichlet allocation is a generative probabilistic topic model representing documents as mixtures of latent topics and topics as distributions over words.
n-gramcontrasts with · An n-gram is a contiguous sequence of n consecutive items, commonly linguistic tokens or characters, drawn from a larger sequence.
spaCyused by · spaCy is an open-source Python library for natural language processing, providing tools and pretrained pipelines for tasks such as tokenization and linguistic analysis.

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.