liveBook · Manning

Map

As explained in Fast Python: High performance techniques for large datasets

A map is an associative data structure that stores key-value pairs and supports retrieving values by key.

A Python dictionary is a map: it associates keys with values so a known key can retrieve its value. In a weather-data pipeline, station identifiers serve as keys and lists of that station’s files serve as values. section 2.4

def download_all_data(stations, start_year, end_year):
    for station in stations:
        for year in range(start_year, end_year + 1):
            if not os.path.exists(TEMPLATE_FILE.format(
             station=station, year=year)):
                download_data(station, year)
 
 
def get_all_files(stations, start_year, end_year):
    all_files = collections.defaultdict(list)
    for station in stations:
        for year in range(start_year, end_year + 1):
            f = open(TEMPLATE_FILE.format(station=station, year=year), 'rb')
            content = list(f.read())
            all_files[station].append(content)
            f.close()
    return all_files
 
 
stations = ['01044099999']
start_year = 2005
end_year = 2021
download_all_data(stations, start_year, end_year)
all_files = get_all_files(stations, start_year, end_year)

The loading code uses `collections.defaultdict(list)` to group file contents by station. Inspect the resulting values as well as the dictionary when assessing memory use. section 2.4

Choose a dictionary when the key is known, not simply because lookups are fast. CPython sets use hashing and are generally implemented like dictionaries without values, so set membership illustrates why dictionary-key lookup is usually fast. That intuition is not a constant-time guarantee. For an interval search, a dictionary would require lookups across the interval; an ordered list with bisection is a better fit. section 2.3.3

Measure the whole structure before optimizing it. `sys.getsizeof(all_files)` measures the outer dictionary, not the lists and file contents it references. A small reported dictionary size can therefore coexist with substantial memory use in its values. section 2.4.1

print(sys.getsizeof(all_files))
print(sys.getsizeof(all_files.values()))
print(sys.getsizeof(list(all_files.values())))

These calls compare the outer dictionary, its values iterator, and a list made from that iterator; none recursively totals the stored file data. section 2.4.1

Key-style access also appears in pandas row computations. In one benchmark, `row["tip_amount"]` outperformed attribute access, but that relationship can change with pandas versions. Benchmark access patterns in the actual workload, then consider whether vectorized computation can avoid row-by-row access altogether. section 7.2.2

How Fast Python: High performance techniques for large datasets covers it

How deep it goes
Explains lookup through its similarity to hashed sets and examines container memory, without developing a dictionary implementation.
Examples
Uses station-to-file mappings, container-size checks, set-versus-list lookup timings, and pandas tip calculations.
What it assumes
Assumes familiarity with Python collections and indexing; develops performance choices through measurements rather than introductory syntax.
What you can do after
Evaluate/understand when keyed lookup fits a task, and diagnose/debug misleading dictionary memory measurements.
How it connects
Connects dictionaries to sets, ordered lists, bisection, memory allocation, and pandas row access.
Cover of Fast Python: High performance techniques for large datasets
liveBookmanning.com
Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Also explained in

Compare how 42 books explain Map →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.