Larger-than-memory data

Q: How deeply does Fast Python: High performance techniques for large datasets cover larger-than-memory data?

Builds from memory mapping and chunking to Dask’s partitioning, lazy semantics, task graphs, and scheduling trade-offs.

As explained in Fast Python: High performance techniques for large datasets

Larger-than-memory data is data whose total size exceeds the available main memory for processing it all at once.

Larger-than-memory data exceeds available main memory, so processing must avoid loading the complete dataset or data structure at once. Two foundational techniques are memory mapping and chunking, applied here to Parquet and CSV files. section 8.3

Dask extends familiar pandas- and NumPy-like interfaces to larger-than-memory data frames and arrays. Its partitioned data model and mostly lazy execution differ from eager in-memory libraries, making the execution model central to effective use. section 10.0 section 10.4

Practical approaches include:

Use memory mapping or chunking as direct techniques for processing oversized files. section 8.3
Use Dask when larger-than-memory arrays or data frames require partitioned computation across a single machine or larger architectures. section 10.0 section 10.4
Inspect task graphs, persistence, repartitioning, and scheduler behavior when analyzing execution and performance trade-offs. section 10.4

From files to partitioned computation

A CSV or Parquet workload can begin with chunking or memory mapping. A larger analysis can instead use a Dask data frame or array, inspect its task graph, and choose an execution architecture appropriate to the environment. section 8.3 section 10.4

How Fast Python: High performance techniques for large datasets covers it

Cover of Fast Python: High performance techniques for large datasets

liveBook manning.com

Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Related concepts

text chunkaddressed by · A text chunk is a bounded segment of a larger text treated as a unit for processing, storage, or retrieval.
datasetkind of · A dataset is an organized collection of related data items, records, or observations treated as a unit for storage, processing, analysis, or computation.

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.