Memory usage is the space occupied by a computation’s data and its representation. It matters before a program runs out of memory: smaller allocations can leave room for parallel processes or keep an algorithm in memory rather than on disk. section 2.4
Measure the representation, not just the input file. A Python container has its own allocation and may refer to many other objects. `sys.getsizeof` reports the container’s size, not the full graph of objects reachable from it. An approximate total must visit contents, count mapping keys and values, and track object identities so shared objects are not counted twice. Even then, native-library objects may not expose enough information for a complete Python-only estimate. section 2.4.1 section 2.4.4
def summarize_columns(df):
for c in df.columns:
print(c, len(df[c].unique()),
df[c].memory_usage(deep=True) // (1024**2), sep="\t")
summarize_columns()
Use this column-by-column pandas check to find expensive columns before changing their types. Deep memory reporting matters when columns contain objects. section 7.1.2
Shrink a taxi-trip table
A loaded taxi-trip DataFrame initially reports about 2 GB. Inspect column types and memory use, parse timestamp strings as datetimes, and choose narrower representations where the values permit them. Missing values complicate integer conversion; a broader type can consume much more space than the underlying values suggest. Reducing a fare column from `float64` to `float32` halves that column’s allocation, but requires an error check. Further type changes bring the reported table footprint to 250.4 MB; that result does not make reduced precision safe for every dataset. section 7.1.2 section 7.1.3
Choose an allocation strategy for the work that follows:
- When results need only one pass, yield them with a generator instead of materializing a list; an eager list can still make sense for repeated visits when its size is tolerable. section 2.5.1
- When loading a DataFrame, omit unneeded columns or specify suitable types in advance rather than accepting every inferred representation. section 7.6
- For GPU work, distinguish host from device memory. Moving data between them costs time; a CuPy array’s operations can stay on the GPU until a transfer back is requested. section 9.1.2 section 9.3.3
