concept PySpark in category data

appears as: PySpark
Mastering Large Datasets with Python: Parallelize and Distribute Your Python Code

This is an excerpt from Manning's book Mastering Large Datasets with Python: Parallelize and Distribute Your Python Code.

With PySpark, we can call Spark’s Scala methods through Python just like we would a normal Python library, by importing the modules and functions we need. For example, we’ll often be using the SparkConf and SparkContext functions to set up our Spark jobs. We’ll talk more about these functions in chapter 9 when we dive into Spark. For now, we can work with them in Python by importing them from PySpark, as shown in the following listing.

Listing 7.6. Importing from Spark into Python
from pyspark import SparkConf, SparkContext

config = SparkConf().setAppName("MyApp")
context = SparkContext(conf=config)

We’ll see this in full force later in this chapter when we dive into PySpark in section 7.5.

In chapter 7, we learned about Hadoop and Spark, two frameworks for distributed computing. In chapter 8, we dove into the weeds of Hadoop, taking a close look at how we might use it to parallelize our Python work for large datasets. In this chapter, we’ll become familiar with PySpark—the Scala-based, in-memory, large dataset processing framework.

Figure 9.6. When we calculate the Elo ratings of tennis players using PySpark, our output will be a sequence of players and their ratings—the higher the rating, the better the player.
Listing 10.6. A robust random forest model using PySpark
from pyspark.ml.classification import RandomForestClassifier
forest = RandomForestClassifier()                             #1
grid = ParamGridBuilder().\                                   #2
           addGrid(forest.maxDepth, [0, 2]).\                 #2
           build()                                            #2
cv = CrossValidator(estimator=forest, estimatorParamMaps=grid,
                    evaluator=bce,numFolds=10,
                    parallelism=4)                            #3
cv_model = cv.fit(df)
area_under_curve = bce.evaluate(cv_model.transform(df))       #4
print("Random Forest AUC: {:0.4f}".format(area_under_curve))
#
print(cv_model.bestModel.toDebugString)                       #5
#