This chapter covers:
- Getting to know fsspec, an abstraction library over file systems
- Efficiently storing heterogeneous columnar data with Parquet
- How to process data files that are larger than memory with in-memory libraries like Pandas or Parquet
- Efficiently process homogeneous multi-dimensional array data with zarr
When dealing with big data, persistence is of paramount importance. We want to be able to access—both read and write—data as fast as possible, preferably from many parallel processes. We also want persistence representations that are compact, as storing large amounts of data can be expensive.
In this chapter we are going to consider several approaches to do persistent storage of data in a more efficient way. We start with a short discussion of fsspec, a library that abstracts access to file systems, local or remote. While this isn’t directly involved in performance issues, it is a modern library used by many applications to deal with storage systems and its use is recurrent in efficient storage implementations.
We then introduce Parquet: a file format to persist heterogeneous columnar datasets. Parquet is supported in Python via the Apache Arrow project which we introduced in the previous chapter.