concurrent.futures provides a declarative, high-level interface for running callable tasks with managed threads or processes. It can replace lower-level threading or multiprocessing code when straightforward task submission matters more than precise worker control. section 3.3.1
A typical pattern uses an executor context, submits mapping work, collects results, and then submits reduction work. ThreadPoolExecutor supports the threaded version; ProcessPoolExecutor offers a process-based alternative, but changing executors is not always drop-in compatible. section 3.3 section 3.4.1
The abstraction is useful, but its execution model imposes practical limits:
- executor.map can make the caller wait for the complete computation, so it provides no immediate progress interaction for long-running workloads. section 3.3.1
- Worker-count management is largely opaque; use lower-level threading when execution must be tuned precisely. section 3.3.1
- Process execution communicates through pickle, preventing direct transfer of lambdas and complicating foreign-library objects, file pointers, database connections, and sockets. section 3.4.1
- Threaded MapReduce can retain all shuffled results in memory, creating a substantial memory footprint even when the representation is compact. section 3.3.1
Threaded MapReduce
A threaded executor maps an emitter over words, groups emitted key-value pairs in a distributor, and maps a reducer over the grouped items. Replacing a lambda reducer with a named def function makes the corresponding process-based example transferable through pickle. section 3.3 section 3.4.1
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor as Executor #1
def map_reduce_still_naive(my_input, mapper, reducer):
with Executor() as executor: #2
map_results = executor.map(mapper, my_input) #3
distributor = defaultdict(list) #4
for key, value in map_results:
distributor[key].append(value)
results = executor.map(reducer, distributor.items()) #3
return results
Runnable listing showing executor context management, map submission, result distribution, and reduction. section 3.3.1
