concept computation in category dask

appears as: computations, computation, computations, computation
Data Science with Python and Dask

This is an excerpt from Manning's book Data Science with Python and Dask.

Medium datasets are datasets that cannot be held entirely in RAM but can fit comfortably in a single computer’s persistent storage. These datasets typically range in size from 10 GB to 2 TB. While it’s possible to use the same toolset to analyze both small datasets and medium datasets, a significant performance penalty is imposed because these tools must use paging in order to avoid out-of-memory errors. These datasets are also large enough that it can make sense to introduce parallelism to cut down processing time. Rather than limiting execution to a single CPU core, dividing the work across all available CPU cores can speed up computations substantially. However, Python was not designed to make sharing work between processes on multicore systems particularly easy. As a result, it can be difficult to take advantage of parallelism within Pandas.

The output of listing 2.1 might not be what you expected. While Pandas would display a sample of the data, when inspecting a Dask DataFrame, we are shown the metadata of the DataFrame. The column names are along the top, and underneath is each column’s respective datatype. Dask tries very hard to intelligently infer datatypes from the data, just as Pandas does. But its ability to do so accurately is limited by the fact that Dask was built to handle medium and large datasets that can’t be loaded into RAM at once. Since Pandas can perform operations entirely in memory, it can quickly and easily scan the entire DataFrame to find the best datatype for each column. Dask, on the other hand, must be able to work just as well with local datasets and large datasets that could be scattered across multiple physical machines in a distributed filesystem. Therefore, Dask DataFrames employ random sampling methods to profile and infer datatypes from a small sample of the data. This works fine if data anomalies, such as letters appearing in a numeric column, are widespread. However, if there’s a single anomalous row among millions or billions of rows, it’s very improbable that the anomalous row would be picked in a random sample. This will lead to Dask picking an incompatible datatype, which will cause errors later on when performing computations. Therefore, a best practice to avoid that situation would be to explicitly set datatypes rather than relying on Dask’s inference process. Even better, storing data in a binary file format that supports explicit data types, such as Parquet, will avoid the issue altogether and bring some additional performance gains to the table as well. We will return to this issue in a later chapter, but for now we will let Dask infer datatypes.

Figure 2.5 The directed acyclic graph representing the computation in listing 2.7

c02_05.eps