Lazy evaluation delays an expression until its result is needed or explicitly requested. In large-data workflows, this lets you describe reading data, filtering rows, creating features, and training a model without immediately loading or computing every step. Dask represents that planned workflow as a directed acyclic graph (DAG). section 8.1
This is useful when a dataset will not fit in RAM. A Dask read can return quickly because it has not yet read the dataset into memory; a filter can likewise be added without immediately processing the data. Computation occurs when a later operation needs the prior results. section 8.1
Figure 8.2 This diagram shows a sample Dask workflow DAG, staring with reading in a dataset, followed by a few steps ending in training a model. Dask can store the steps in a workflow like this as a DAG. When a step in the DAG needs to output the results from previous steps, then the preceding steps in the DAG workflow will be executed.
Follow the DAG from dataset reading through filtering and feature creation to model training. The arrows show dependencies: requesting an output causes the preceding required steps to run. section 8.1
Request evaluation deliberately when you need an answer:
- In Dask, append compute() to calculate and return a result, such as mean click rate by site category. section 8.1.1
- In PySpark, use collect() to return a result for later use, or show() to calculate and print grouped output. section 8.2.3
- Leave off these calls while you are still composing transformations and do not need their results yet. section 8.1.1 section 8.2.3
Calculate click rate only when requested
A Dask workflow can read selected ad-data columns, filter to clicks, group by site_category, and take the mean. Adding compute() to the end asks for the grouped result at once. Without it, the grouped calculation remains deferred. Because click is binary, its mean gives the proportion of clicks for each category. section 8.1.1
Dask and PySpark pair lazy evaluation with parallel processing. They can use available CPU cores on one machine and can scale work across multiple machines, helping make some large-data operations feasible while saving memory resources and potentially compute time. section 8.1.5 section 8.5















