concept microblog text in category solr
appears as: microblog text, microblog text

This is an excerpt from Manning's book Solr in Action.
Table 6.2. Overview of the transformations made to the microblog text using Solr’s text analysis tools. Notice how all transformations are performed using built-in Solr text analysis tools: no custom code required!
Raw text
Transformation
How we did it with Solr
All terms Lowercased (SF's → sf's) LowerCaseFilterFactory a, at, in, with, by, on Removed from text Very common terms called “stop words” removed from the text using StopFilterFactory Drinking, learning drink, learn Stemming with KStemFilterFactory SF's sf, san francisco Apostrophes ('s) removed by WordDelimiterFilterFactory Caffé caffe Diacritic é transformed to e using ASCIIFoldingFilterFactory i-Pad ipad, i pad Hyphenated word correctly handled using WordDelimiterFilterFactory #Yummm #yumm Collapse repeated letters down to a maximum of two using PatternReplaceCharFilterFactory #SolrInAction, @ManningBooks #solrinaction, @manningbooks Hashtags and mentions correctly preserved using the WhitespaceTokenizer and WordDelimiterFilterFactory