A character filter cleans a text field before tokenization. It is the optional first stage of an Elasticsearch analyzer: it can remove unwanted characters, replace characters with text, or add text to the incoming character stream. The cleaned result then goes to one tokenizer, followed by zero or more token filters. section 7.2.3 section 7.6
Use a character filter when raw input contains text that should not reach tokens or the inverted index:
- Strip markup such as <h1>Where is my cheese?</h1> so users can search the visible words rather than HTML or XML tags. section 7.6
- Map symbols or values to searchable words, such as β to beta or 0 and 1 to false and true. section 7.6
- Apply regular-expression matching and replacement, such as extracting an organization domain from an email address. section 7.2.3
Elasticsearch supplies html_strip, mapping, and pattern_replace character filters. An analyzer may use none or several of them, but it must use exactly one tokenizer. Analysis runs during indexing and also when search queries are analyzed, so cleaning rules belong in the analyzer configuration rather than in ad hoc input handling. section 7.2.3 section 7.9
Strip HTML before tokenizing
Define a custom analyzer with html_strip, then test it: PUT index_with_custom_analyzer { "settings": { "analysis": { "analyzer": { "custom_analyzer": { "type": "custom", "char_filter": ["html_strip"], "tokenizer": "standard", "filter": ["uppercase"] } } } } } POST index_with_custom_analyzer/_analyze { "text": "<H1>HELLO, WoRLD</H1>", "analyzer": "custom_analyzer" } html_strip removes the H1 tags first. The standard tokenizer then splits the remaining text, and uppercase produces HELLO and WORLD. section 7.4
Listing 7.24 Custom analyzer with filters and a tokenizer
PUT index_with_custom_analyzer { "settings": { "analysis": { "analyzer": { "custom_analyzer":{ #1 "type":"custom", #2 "char_filter":["html_strip"], #3 "tokenizer":"standard", #4 "filter":["uppercase"] #5 } } } } } #1 Specifies the custom analyzer #2 The type must be custom to let Elasticsearch know about our custom analyzer. #3 Array of character filters #4 Declares a single tokenizer (standard, in this case) #5 Token filter to uppercase incoming tokens
Inspect the custom-analyzer request to see where the character-filter array sits relative to the required tokenizer and token-filter array. section 7.4
