An executor manages which coroutine runs next. Coroutines yield voluntarily at await points, allowing another task to run while the first waits; this is cooperative scheduling, unlike the preemptive scheduling of threads. section 3.1.2
In concurrent.futures, an executor manages threads and can run mapper and reducer tasks. Its worker management is largely a black box, and map waits for results; submit returns futures that can be checked for completion. section 3.3.1 section 3.3.2
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor as Executor #1
def map_reduce_still_naive(my_input, mapper, reducer):
with Executor() as executor: #2
map_results = executor.map(mapper, my_input) #3
distributor = defaultdict(list) #4
for key, value in map_results:
distributor[key].append(value)
results = executor.map(reducer, distributor.items()) #3
return results
See a ThreadPoolExecutor used to run map and reduce tasks in a word-count example. section 3.3.1
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor as Executor
def async_map(executor, mapper, data):
futures = []
for datum in data:
futures.append(executor.submit(mapper, datum)) #1
return futures
def map_less_naive(executor, my_input, mapper):
map_results = async_map(executor, mapper, my_input)
return map_results
See tasks submitted individually so their futures can be tracked before results are collected. section 3.3.2
Futures support completion checks and progress reporting through callbacks. Callbacks should be fast because other work waits while they run; the simple threaded implementation also retains intermediate results in memory. section 3.3.2 section 3.3.1


