liveBook · Manning

Executor

As explained in Fast Python: High performance techniques for large datasets

An executor is a software component or interface that accepts tasks or actions and arranges for their execution by an appropriate runtime or worker.

An executor manages which coroutine runs next. Coroutines yield voluntarily at await points, allowing another task to run while the first waits; this is cooperative scheduling, unlike the preemptive scheduling of threads. section 3.1.2

In concurrent.futures, an executor manages threads and can run mapper and reducer tasks. Its worker management is largely a black box, and map waits for results; submit returns futures that can be checked for completion. section 3.3.1 section 3.3.2

from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor as Executor      #1
 
def map_reduce_still_naive(my_input, mapper, reducer):
    with Executor() as executor:                                   #2
        map_results = executor.map(mapper, my_input)               #3
 
        distributor = defaultdict(list)                            #4
        for key, value in map_results:
            distributor[key].append(value)
 
        results = executor.map(reducer, distributor.items())       #3
    return results

See a ThreadPoolExecutor used to run map and reduce tasks in a word-count example. section 3.3.1

from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor as Executor
 
def async_map(executor, mapper, data):
    futures = []
    for datum in data:
        futures.append(executor.submit(mapper, datum))   #1
    return futures
 
def map_less_naive(executor, my_input, mapper):
    map_results = async_map(executor, mapper, my_input)
    return map_results

See tasks submitted individually so their futures can be tracked before results are collected. section 3.3.2

Futures support completion checks and progress reporting through callbacks. Callbacks should be fast because other work waits while they run; the simple threaded implementation also retains intermediate results in memory. section 3.3.2 section 3.3.1

How Fast Python: High performance techniques for large datasets covers it

How deep it goes
Defines cooperative scheduling and demonstrates thread pools, futures, and callbacks, but leaves worker tuning largely opaque.
Examples
Uses network and disk coroutines plus threaded word-count mapping and reduction to illustrate execution.
What you can do after
Apply ThreadPoolExecutor map or submit to run tasks and track their completion with futures and callbacks.
How it connects
Connects executors to coroutines, cooperative scheduling, futures, callbacks, and MapReduce.

Also explained in

Compare how 3 books explain executor →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.