Our journey with PySpark thus far has proven that it is a powerful and versatile data-processing tool. So far, we’ve explored many out-of-the-box functions and methods to manipulate data in a data frame. Recall from chapter 1 that PySpark’s data frame manipulation functionality takes our Python code and applies an optimized query plan. This makes our data jobs efficient, consistent, and predictable, just like coloring within the lines. What if we need to go off-script and manipulate our data according to our own rules?