concept microblog text in category solr

appears as: microblog text, microblog text
Solr in Action

This is an excerpt from Manning's book Solr in Action.

Table 6.2. Overview of the transformations made to the microblog text using Solr’s text analysis tools. Notice how all transformations are performed using built-in Solr text analysis tools: no custom code required!

Raw text

Transformation

How we did it with Solr

All terms Lowercased (SF's → sf's) LowerCaseFilterFactory
a, at, in, with, by, on Removed from text Very common terms called “stop words” removed from the text using StopFilterFactory
Drinking, learning drink, learn Stemming with KStemFilterFactory
SF's sf, san francisco Apostrophes ('s) removed by WordDelimiterFilterFactory
Caffé caffe Diacritic é transformed to e using ASCIIFoldingFilterFactory
i-Pad ipad, i pad Hyphenated word correctly handled using WordDelimiterFilterFactory
#Yummm #yumm Collapse repeated letters down to a maximum of two using PatternReplaceCharFilterFactory
#SolrInAction, @ManningBooks #solrinaction, @manningbooks Hashtags and mentions correctly preserved using the WhitespaceTokenizer and WordDelimiterFilterFactory
Listing 6.2. Custom field type for analyzing microblog text