Lazy evaluation delays computation until a result is needed. For large data pipelines, this can spread work and memory allocation over time—or avoid work entirely when only part of the result is required. section 2.5
Process temperatures one at a time
A CSV-reading function can yield each valid temperature instead of returning a list. Calling it creates a generator; iteration runs the parsing code as values are needed. A search for any temperature below zero can stop at the first match, without producing the remaining values. Converting the generator to a list gives up those savings but can make sense when the results fit in memory and must be visited repeatedly. section 2.5.1
def get_file_temperatures(file_name): with open(file_name, "rt") as f: reader = csv.reader(f) header = next(reader) for row in reader: station = row[header.index("STATION")] # date = datetime.datetime.fromisoformat( row[header.index('DATE')]) tmp = row[header.index("TMP")] temperature, status = tmp.split(",") if status != "1": continue temperature = int(temperature) / 10 yield temperature #1 #1 A yield in a definition indicates a generator.
The yield makes the CSV reader a generator that produces valid temperatures one at a time. section 2.5.1
Measure the operation you intend to optimize. In one %timeit comparison, creating range(1000000) took about 189 ns, versus 27.4 ms for list(range(1000000)). That comparison measures constructing the lazy range against constructing a populated list; it does not measure consuming every range value. section 2.2.3
Dask applies laziness at the task-graph level. Data-frame operations build a graph, and compute() executes it and returns a pandas data frame. This differs from pandas or NumPy operations that compute when specified. Inspecting the graph can help assess planned work before execution, but do not assume every Dask operation is deferred: set_index can trigger computation, and repartitioning can be expensive. section 10.1.2 section 10.2.4
import numpy as np
import dask.dataframe as dd #1
taxes = dd.read_csv("FY2016-STC-Category-Table.csv", sep="\t")
taxes["Amount"] = taxes["Amount"].str.replace(",",
"").replace("X", np.nan).astype(float)
taxes["Tax_Type"] = taxes["Tax_Type"].astype(
"category").cat.as_known() #2
pivot = taxes.pivot_table(index="Geo_Name",
columns="Tax_Type", values="Amount")
has_property_info = pivot[
~pivot["Property Taxes"].isna()].index #3
pivot_clean = pivot.loc[has_property_info]
frac_property = pivot_clean["Property Taxes"] / pivot_clean["Total Taxes"]
The data-frame pipeline specifies CSV loading, cleaning, and a property-tax calculation before requesting a result. section 10.1.2
