In-memory processing

Q: How deeply does Fast Python: High performance techniques for large datasets cover in-memory processing?

Explains allocation effects, memory estimation limits, and scaling trade-offs, but stops short of prescribing a complete storage architecture.

As explained in Fast Python: High performance techniques for large datasets

In-memory processing performs computations primarily on data held in main memory rather than repeatedly accessing persistent storage.

In-memory processing keeps working data in main memory instead of repeatedly relying on disk storage. It can greatly reduce access costs, but only while the dataset fits available memory. section 2.4 section 1.1

Use the technique with attention to:

Reducing allocations can leave enough memory for more parallel processes and make an in-memory algorithm possible. section 2.4.5
Measure memory as well as CPU time; an implementation that spills computation to disk may be much slower. section 2.6 section 5.6.2
Expect a redesign when data outgrows memory, potentially requiring caching, concurrent reads, or changed storage technology. section 1.1

Loading NOAA files

A straightforward Python implementation downloads station-year files, reads each file into a list of bytes, and stores all contents in a dictionary. Inspecting this structure exposes its memory requirements and creates a baseline for reducing disk use. section 2.4

Game of Life benchmark

A memory-intensive Game of Life implementation runs substantially faster after moving critical computation from Python to Cython. The comparison demonstrates the value of in-memory work while also warning that resolution can make memory consumption dangerous. section 5.6.2

def download_all_data(stations, start_year, end_year):
    for station in stations:
        for year in range(start_year, end_year + 1):
            if not os.path.exists(TEMPLATE_FILE.format(
             station=station, year=year)):
                download_data(station, year)
 
 
def get_all_files(stations, start_year, end_year):
    all_files = collections.defaultdict(list)
    for station in stations:
        for year in range(start_year, end_year + 1):
            f = open(TEMPLATE_FILE.format(station=station, year=year), 'rb')
            content = list(f.read())
            all_files[station].append(content)
            f.close()
    return all_files
 
 
stations = ['01044099999']
start_year = 2005
end_year = 2021
download_all_data(stations, start_year, end_year)
all_files = get_all_files(stations, start_year, end_year)

Inspect the file-loading listing as a concrete baseline for accumulating station data in memory. section 2.4

How Fast Python: High performance techniques for large datasets covers it

Cover of Fast Python: High performance techniques for large datasets

liveBook manning.com

Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Related concepts

DuckDBuses · DuckDB is an open-source, in-process analytical database management system with a columnar-vectorized execution engine for fast local querying.
memory allocationproduced by · Memory allocation is the process of reserving or assigning storage for program data and objects during execution.
machine learningimplements · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
cachingused by · Caching is the practice of storing reusable data or computation results so later access can avoid repeating retrieval or computation.

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.