Parallelism means tasks execute at the same time, rather than merely taking turns. A multicore CPU can run independent work simultaneously, but sequential Python code does not use those cores automatically. Concurrency is different: tasks may overlap in time without executing simultaneously, which can still be useful when they spend time waiting for I/O. section 1.3.1 section 3.0
Start by identifying work that can be divided into sufficiently large, independent tasks. Then choose an execution model that fits the work and measure its costs. Adding workers is not a guarantee of speedup: communication, synchronization, and stages that must run as one task can erase the benefit. section 3.6 section 10.2.3
In Python, the route to simultaneous execution depends on where the computation runs:
- For pure Python CPU work, use multiprocessing to run work across CPU cores. Keep tasks coarse-grained: frequent communication between processes can become the bottleneck. section 3.4 section 3.6
- Do not assume CPython threads will speed up CPU-bound Python code. The Global Interpreter Lock (GIL) permits only one thread to execute Python code at a time. Lower-level code can release it; Cython can use an OpenMP-backed parallel range after GIL-dependent operations have been removed. section 3.3.3 section 5.7
- Use GPU execution for large numbers of similar operations, such as applying the same computation across array elements. GPU code needs a different execution approach from sequential CPU code; simply annotating existing Python code is not enough. section 1.2.1 section 9.4
- Use Dask when work must scale across partitions on one computer or across machines. Inspect its task graph: independent partition operations can run together, while reductions, sorting, and data transfers can restrict parallelism. section 10.0 section 10.2.3
From a correct MapReduce result to parallel execution
A straightforward word counter maps each word to a key and value, groups values by key, and reduces each group to a count. It produces the expected result, but its map calls are not parallel merely because it uses map(). Running CPU-bound mapping through multiprocessing addresses that limitation; the size of each unit of work and the cost of moving results between processes still matter. section 3.2.3 section 3.6
Where a distributed computation narrows
Dividing each partition’s amounts by 1,000 lets Dask compute partitions independently. Finding the overall maximum first computes a maximum per partition, then gathers those results for a final maximum. That final task reduces the available parallelism. Render the task graph with visualize() to find such bottlenecks before assuming a distributed operation will scale well. section 10.2.3
Parallel code also has a reliability cost. Races, deadlocks, and nondeterministic communication can make failures difficult to reproduce. Prefer simple divisions of work over shared memory and low-level locks; optimize only where the expected gain justifies the added complexity. section 1.1 section 3.6
