liveBook · Manning

Persist

As explained in Fast Python: High performance techniques for large datasets

persist is a Dask collection operation that starts computation and retains the resulting partitions for reuse in subsequent computations.

persist is a Dask collection operation that starts computing a task graph and retains the resulting partitions for reuse. Its exact execution semantics depend on the scheduler. section 10.2.2

Use persist when repeated computation or data movement is more expensive than retaining partitions:

  • Persist frequently reused graph nodes, especially when they produce relatively small amounts of data. section 10.2.2
  • Keep partitions distributed so later queries can run in parallel without gathering, repartitioning, and redistributing the complete collection. section 10.2.2
  • Avoid persisting everything: retained partitions consume memory, while transferring data can incur costly serialization, particularly over a network. section 10.2.2
  • Do not confuse Dask's .persist method with writing data to persistent storage such as disk, CSV, or Parquet. section 10.2.5

Retaining a parsed Dask column

After removing commas and invalid values from taxes["Amount"] and converting the column to floats, call taxes = taxes.persist(). Subsequent tax queries reuse the computed partitions instead of repeating the string conversion. section 10.2.2

taxes["Amount"] = taxes["Amount"].str.replace(",",
  "").replace("X", np.nan).astype(float)
taxes = taxes.persist()
taxes.visualize(filename="10-persist.svg", rankdir="LR")

Inspect the compact Dask workflow that parses the Amount column, persists the collection, and visualizes the resulting task graph. section 10.2.2

from time import time
 
def time_scenario(size, persist_range, persist_pos, chunk_div=10):
    start_time = time()
    size = size
    range_array = da.arange(0, size*size).reshape(size, size).persist()
    range_array = range_array.rechunk(size // chunk_div, size // chunk_div)
    range_array = range_array.persist() if persist_range else range_array
    pos_array = da.blockwise(
        lambda x: block_prepare_pos_array(size, x),
        'ij', range_array, 'ij', dtype=np.complex128)
 
    pos_array = pos_array.persist() if persist_pos else pos_array
    image_arr = u_compute_point(pos_array)
    image_arr.visualize("task_graph.png", rankdir="TB")
    image_arr.compute()
    return time() - start_time

Compare execution time with and without persisting range and position arrays while computing a chunked Mandelbrot image. section 10.3.2

How Fast Python: High performance techniques for large datasets covers it

How deep it goes
Explains computation, partition retention, scheduler dependence, memory limits, and network costs, but stops short of scheduler internals.
Examples
Uses parsed tax amounts, task-graph visualization, and a chunked Mandelbrot timing experiment.
What it assumes
Assumes familiarity with Dask collections, task graphs, partitions, and compute; teaches the distinction between retained partitions and disk persistence.
What you can do after
Apply it via Dask to retain reusable partitions and benchmark whether reduced recomputation outweighs memory consumption.
How it connects
Connects Dask collections with task graphs, partitions, schedulers, compute, serialization, and distributed computing.
Cover of Fast Python: High performance techniques for large datasets
liveBookmanning.com
Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.