This chapter covers:
- How to use the RDD as a low level, flexible data container.
- How to promote regular Python functions to UDF to run in a distributed fashion.
- How to use scalar UDF as an alternative to Python UDF, using pandas' API.
- How to use grouped map and grouped aggregate UDF on
GroupedDataobject to split data frame computation on manageable chunks.
- How to apply UDF on local data to ease debugging.
Our journey with PySpark so far has proven that it is a powerful and versatile data processing tool. So far, we’ve explored many out-of-the-box functions and methods to manipulate data in a data frame. PySpark’s data frame manipulation functionality takes our Python code and applies an optimized query plan, introduced in Chapter 1. This makes our data jobs efficient, consistent, and predictable, just like coloring within the lines. What if we need to go off-script and manipulate our data according to our own rules?
In this chapter, I cover how we can build Python functions and scale them in PySpark. I start by introducing the resilient distributed dataset (or RDD), a more primitive and lower-level structure compared to the data frame. I explain how you manipulate data in an RDD and how its element (or row) major nature complements the data frame column-major approach.