Performance optimization starts with a workload and a constraint, not a faster-looking line of code. Growing data may exceed memory or force work onto multiple machines, where partitioning and coordination add complexity. Inspect the system’s architecture and implementation before assuming that more servers will help. section 1.1 section 1.2
For an existing Python pipeline, profile before changing it. CPU time is an obvious starting point, but memory use and I/O can determine which design is viable. Guessing at bottlenecks is unreliable: measure the baseline, locate expensive work, change one approach, and measure again. section 2.0 section 2.6
Choose a change that addresses the measured cost:
- Inspect operations on built-in data structures. Repeated searches through an unordered list can be expensive; consider operation complexity rather than treating containers as interchangeable. section 2.6
- Reduce the memory footprint when it affects execution. Lazy techniques can avoid storing data—or avoid parts of a computation altogether—and fitting work in memory can yield large time gains. section 2.6
- Account for the execution environment. Cloud resource costs, virtualization, and storage APIs can change the value of an optimization that works on a single machine. section 1.2.3
A constrained ingestion pipeline
Suppose a climate-data ingestion pipeline must be improved before a large dataset arrives, with mostly standard Python and no budget for more processing power. Profile the existing code first, then focus effort where the profile identifies a bottleneck rather than rewriting the pipeline on intuition. section 2.0
Distributed work needs a different inspection point: the task graph. In Dask, operations that require communication or reduce parallelism can be costly, while some partition-local computations do not communicate. Inspect the graph for the actual operations and environment; costs and optimizations can change across architectures and Dask versions. section 10.2.3
sv = taxes.sort_values("k_amount")
sv.visualize(filename="10-sv.svg", rankdir="LR")
Visualizing a Dask sort makes its task structure inspectable before judging where distributed execution may be expensive. section 10.2.3
