8 Alternatives to Pandas
This chapter covers
- Parallel machine learning training with Dask
- Exploratory data analysis and processing with PySpark
- Machine learning models training with PySpark
- Other alternatives to pandas
In the last two chapters, we’ve discussed scaling Python code in terms of both computational speed and dealing with memory challenges. However, many of the examples in Chapters 6 and 7 still rely largely on standard data science packages like pandas or sklearn. In this chapter, we will focus on taking scaling to the next level by learning about packages that can distribute many of the tasks in pandas or scikit-learn across either a single machine or a cluster of machines. This will allow for both computational speed improvements in terms of parallelization, and also the ability to process very large datasets by scaling the appropriate number of machines you need by the size of your data. The two primary packages we’ll focus on are Dask and PySpark. We will also touch on a few others near the end of the chapter. There are several reasons why we need to consider moving beyond standard pandas or scikit-learn: