A token is a searchable unit produced from text, such as a word, phrase, or symbol. Tokenization lets a search index work with parts of a string rather than only the complete string. An inverted index associates those units with documents so a query can find matching documents quickly. section 11.3.3 section 11.2
An analyzer determines how text becomes searchable terms. Its tokenizer segments the text; filters then adjust the resulting tokens for variations such as punctuation, capitalization, and unnecessary words. The same choice matters for query text: Atlas Search lets you specify an analyzer for indexing and a search analyzer for queries. If you specify neither, it uses the standard analyzer. section 11.3.3
Matching inspection records
Two inspection documents contain the values “No Violation Issued” and “Mobile Food Vendor - 881.” In a simplified inverted index, each value is a searchable unit associated with both documents. A query using those units can therefore find both records. This example uses multiword units; the tokens produced in an actual index depend on its analyzer. section 11.2 section 11.3.3
Table 11.3 Atlas Search built-in analyzers
| Analyzer |
Description |
| Standard |
Uses the default analyzer for all Atlas Search indexes and queries |
| Simple |
Divides text into searchable terms wherever it finds a nonletter character |
| Whitespace |
Divides text into searchable terms wherever it finds a whitespace character |
| Language |
Provides a set of language-specific text analyzers |
| Keyword |
Indexes text fields as single terms |
Compare built-in analyzers before choosing token boundaries: whitespace splits at spaces, simple splits at nonletters, and keyword indexes a text field as one term. section 11.3.3
Listing 11.1 Atlas Search index syntax
{
"analyzer": "<analyzer-for-index>",
"searchAnalyzer": "<analyzer-for-query>",
"mappings": {
"dynamic": <boolean>,
"fields": { <field-definition> }
},
"analyzers": [ <custom-analyzer> ],
"storedSource": <boolean> | {
<stored-source-definition>
},
"synonyms": [
{
"name": "<synonym-mapping-name>",
"source": {
"collection": "<source-collection-name>"
},
"analyzer": "<synonym-mapping-analyzer>"
}
]
}
Use the index definition’s analyzer and searchAnalyzer fields when indexing text and processing query text call for different analyzer choices. section 11.3.3
