Chapter 11. Taking clustering to production
This chapter covers
- Running a clustering job on a Hadoop cluster
- Tuning a clustering job for performance
- Batch clustering versus online clustering
You’ve seen how different clustering algorithms in Mahout group the documents in the Reuters news data set. Along the way, you learned about the vector representation of data, distance measures, and various other ways to improve the quality of clusters. One of Mahout’s strengths is its ability to scale. The Reuters data set wasn’t much of a challenge, so in this chapter we set a bigger challenge for Mahout: clustering one of the largest free data sets in the world: Wikipedia—the free encyclopedia. Mahout can handle such scales because the algorithms in it are implemented as MapReduce jobs that can execute on a Hadoop cluster over hundreds and thousands of computers.[1]
1 Ajay Anand describes running a 4,000 node Hadoop installation at Yahoo! in 2008 in his blog entry, “Scaling Hadoop to 4000 nodes at Yahoo!,” http://developer.yahoo.net/blogs/hadoop/2008/09/scaling_hadoop_to_4000_nodes_a.html