liveBook · Manning

Token_count data type

As explained in Elasticsearch in Action, Second Edition

The token_count data type is an Elasticsearch mapping type that stores the number of tokens produced for a field.

The token_count data type is a specialized text mapping that stores how many tokens an analyzer produces for a field. A book title mapped this way can be searched by token count, such as finding titles longer than three words. section 4.5.1

A practical token_count mapping follows this pattern:

  • Create an index whose title field has type token_count. section 4.5.1
  • Provide an analyzer; the example uses the standard analyzer to produce the counted tokens. section 4.5.1
  • Index titles such as “Elasticsearch in Action,” “Elasticsearch for Java Developers,” and “Elastic Stack in Action.” section 4.5.1
  • Run a range query to select counts greater than three and less than or equal to five; four-token titles match while a three-token title does not. section 4.5.1

Count titles directly or alongside searchable text

A field can be mapped only as token_count, or a text field can expose an additional word_count subfield with token_count. The latter preserves normal text searching while enabling an exact term query such as title.word_count = 4. section 4.5.1

Listing 4.9 Index with a token_count data type for the title field
PUT tech_books
{
  "mappings": {
    "properties": {
      "title": {                  #1
        "type": "token_count",    #2
        "analyzer": "standard"    #3
      }
    }
  }
}

Inspect the runnable mapping that assigns token_count and the standard analyzer to the title field. section 4.5.1

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Defines the mapping, analyzer requirement, indexing examples, range filtering, and a text multi-field variant.
Examples
Uses technical-book titles, a standard-analyzer mapping, a three-to-five-token range, and an exact four-token multi-field query.
What it assumes
Assumes mapping definitions and basic data types, while requiring the field’s analyzer to be specified.
What you can do after
Operate/configure Elasticsearch REST mappings and queries that filter documents by analyzer-generated token counts.
How it connects
Connects token_count with text fields, analyzers, standard analysis, multi-fields, range queries, and term queries.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.