Word boundary folding gives punctuation and whitespace at word boundaries one uniform character representation. This keeps tokenization consistent when a search pipeline builds character trigrams. section 10.3.1
Consistent boundary trigrams
A trigram index for “trigram index” includes boundary trigrams such as “ tr” and “am ”. Folding every non-alpha boundary character to a space makes punctuation and whitespace produce comparable representations across documents. section 10.3.1
Use the folded representation while creating trigram indices and text vectors. Shared trigrams support approximate matching and more accurate relevance ranking, complementing or replacing stemming, lemmatization, and case folding in some search pipelines. section 10.3.1
