Memory use affects more than whether a program fits in RAM: a smaller footprint can leave room for parallel processes or in-memory algorithms. section 2.4 section 2.4.5
def get_all_files_clean(stations, start_year, end_year):
all_files = collections.defaultdict(list)
for station in stations:
for year in range(start_year, end_year + 1):
f = open(TEMPLATE_FILE.format(station=station, year=year), 'rb')
content = f.read() #1
all_files[station].append(content)
f.close()
return all_files
Reading each file directly as bytes avoids expanding it into a Python list of individual byte values, reducing object overhead. section 2.4.3
Estimating usage requires counting both containers and their contents. In a NOAA data example, a 1,303,981-byte file represented as a list occupies an estimated 46,943,372 bytes. But summing every apparent object size can overcount: CPython may reuse objects, so identity matters. section 2.4.1 section 2.4.1
Listing 2.2 Computing the size of general Python objects
from array import array
from collections.abc import Iterable, Mapping
from sys import getsizeof
from types import GeneratorType
def compute_allocation(obj):
my_ids = set([id(obj)]) #1
to_compute = [obj]
allocation_size = 0
container_allocation = 0 #2
while len(to_compute) > 0:
obj_to_check = to_compute.pop()
allocation_size += getsizeof(obj_to_check)
if type(obj_to_check) in [str, array]: #3
continue
elif isinstance(obj_to_check, GeneratorType): #4
continue
elif isinstance(obj_to_check, Mapping): #5
container_allocation += getsizeof(obj_to_check)
for ikey, ivalue in obj_to_check.items():
if id(ikey) not in my_ids:
my_ids.add(id(ikey))
to_compute.append(id(ikey))
if id(ivalue) not in my_ids:
my_ids.add(ivalue)
to_compute.append(id(ivalue))
elif isinstance(obj_to_check, Iterable): #6
container_allocation += getsizeof(obj_to_check)
for inner in obj_to_check:
if id(inner) not in my_ids:
my_ids.add(id(inner))
to_compute.append(inner)
return allocation_size, allocation_size - container_allocation
This iterative estimator tracks object identities, handles mapping keys and values, and separates total size from container overhead. It is an approximation, not a universal measure. section 2.4.4
sys.getsizeof does not report the full footprint of referenced objects. General estimation has limits, especially for external libraries whose low-level allocations may not be exposed. section 2.4.5 section 2.4.4
