An executor coordinates when tasks run. In asynchronous code, it schedules coroutines that voluntarily yield control; in concurrent.futures, it manages worker threads that execute submitted tasks. section 3.1.2 section 3.3.1
The execution model determines how work progresses and how results become available:
- Coroutines yield at await points, letting the executor switch tasks while operations such as network waits or disk writes are pending. section 3.1.2
- With concurrent.futures, executor.map waits for results, while submit returns futures that can be checked for completion and used to report progress. section 3.3.1 section 3.3.2
- ThreadPoolExecutor simplifies worker management, but fine-tuning workers requires a lower-level interface such as threading. section 3.3.1
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor as Executor #1
def map_reduce_still_naive(my_input, mapper, reducer):
with Executor() as executor: #2
map_results = executor.map(mapper, my_input) #3
distributor = defaultdict(list) #4
for key, value in map_results:
distributor[key].append(value)
results = executor.map(reducer, distributor.items()) #3
return results
This example uses ThreadPoolExecutor to map and reduce word-count tasks. section 3.3.1
