Tokenization divides text into searchable units called tokens, such as words, phrases, or symbols. In Atlas Search, an analyzer processes text when it is indexed or queried. The resulting index terms let a search index locate matching documents without scanning every document. section 11.3.3 section 11.2
Choose how text becomes searchable terms:
- Select a tokenizer to split the text. The built-in whitespace analyzer splits at whitespace, while the simple analyzer splits at nonletter characters. section 11.3.3
- Use analyzer filters to account for variations such as punctuation, capitalization, and unnecessary words. Normalizing text to lowercase can make differently capitalized text searchable as the same term. section 11.3.3 section 11.2
- Specify an analyzer for indexed fields and, if needed, a searchAnalyzer for query text. Atlas Search uses its standard analyzer when none is specified; custom analyzers can combine built-in character filters, tokenizers, and token filters. section 11.3.3
Matching inspection records
In a simplified inverted index, two inspection records share the terms “NO VIOLATION ISSUED” and “MOBILE FOOD VENDOR - 881.” A query matched against those terms returns both records. Their relevance scores can still differ: matching terms that occur frequently in a document gain weight, while terms common across the collection gain less. section 11.2
Table 11.3 Atlas Search built-in analyzers
| Analyzer |
Description |
| Standard |
Uses the default analyzer for all Atlas Search indexes and queries |
| Simple |
Divides text into searchable terms wherever it finds a nonletter character |
| Whitespace |
Divides text into searchable terms wherever it finds a whitespace character |
| Language |
Provides a set of language-specific text analyzers |
| Keyword |
Indexes text fields as single terms |
Compare the built-in analyzers before choosing how to split a field: whitespace and simple split differently, while keyword indexes a text field as one term. section 11.3.3
Listing 11.1 Atlas Search index syntax
{
"analyzer": "<analyzer-for-index>",
"searchAnalyzer": "<analyzer-for-query>",
"mappings": {
"dynamic": <boolean>,
"fields": { <field-definition> }
},
"analyzers": [ <custom-analyzer> ],
"storedSource": <boolean> | {
<stored-source-definition>
},
"synonyms": [
{
"name": "<synonym-mapping-name>",
"source": {
"collection": "<source-collection-name>"
},
"analyzer": "<synonym-mapping-analyzer>"
}
]
}
The index definition shows where to set the indexing analyzer, the query analyzer, and custom analyzers. section 11.3.3
