10 Faster decision making with machine learning and PySpark
This chapter covers
- An introduction to machine learning
- Training and applying decision trees classifiers in parallel with PySpark
- Matching problems and appropriate machine learning algorithms
- Training and applying random forest regressors with PySpark
In chapter 9 we saw some of Spark’s raw data transformation options and we used it in the Map and Reduce style we’ve been exploring throughout the book. Chapter 9 showed how we can write Python and take advantage of Spark, one of the most popular distributed computing frameworks. One of the reasons Spark is so popular is because of its built-in machine learning capabilities.
Machine learning refers to the design, training, application, and study of judgmental algorithms which adjust themselves based on input data. A familiar example of machine learning is the spam filter. Spam filter designers feed spam into their spam filter algorithms, which either are or contain machine learning algorithms. Then the spam filter algorithm learns to make judgments about whether or not an email is spam.
Figure 10.1 Spam filters are machine learning algorithms which learn how to judge emails as spam or not by looking at lots of spam emails and not-spam emails.
