Generators produce values on demand: computation runs as values are requested through iteration, rather than building the full result in advance. This can reduce memory use and avoid work when only some results are needed. section 2.5 section 2.5.1
Choose between lazy generation and eager materialization according to how the results will be used:
- Use a generator for large sequences or when processing can stop early; it avoids holding every result in memory. section 2.5.1
- Convert results to a list when they are small enough to fit in memory and need to be visited repeatedly. section 2.5.1
Reading temperatures from a file
A function that reads CSV rows can yield each valid temperature as it is processed, instead of accumulating all temperatures in a list. section 2.5.1
def get_file_temperatures(file_name): with open(file_name, "rt") as f: reader = csv.reader(f) header = next(reader) for row in reader: station = row[header.index("STATION")] # date = datetime.datetime.fromisoformat( row[header.index('DATE')]) tmp = row[header.index("TMP")] temperature, status = tmp.split(",") if status != "1": continue temperature = int(temperature) / 10 yield temperature #1 #1 A yield in a definition indicates a generator.
See the generator function: yield returns each parsed temperature on demand. section 2.5.1
Generators also support chunked data workflows. With pandas, read_csv can return an iterable of bounded-size data-frame chunks, which can then be converted and written one at a time. section 8.3.2
import pandas as pd
table_chunks = pd.read_csv(
"../../07-pandas/sec1-intro/yellow_tripdata_2020-01.csv.gz",
chunksize=1000000
)
print(type(table_chunks)) #1
for chunk in table_chunks: #2
print(chunk.shape)
This example reads CSV data in chunks of up to one million rows. section 8.3.2
