MapReduce separates a computation into mapping input items and reducing the results. Between those steps, a shuffle groups values by key so that one reducer receives every value for that key. This version processes in-memory data on a single computer. section 3.2.1 section 3.0
Count words
For “I am a fool. To weep at what I am glad of,” the mapper emits a pair such as ("am", 1) for each word. The shuffle brings both "am" values together; the reducer sums them to produce ("am", 2). Sending those values to separate reducers would leave two incorrect counts of 1. section 3.2.1
from collections import defaultdict
def map_reduce_ultra_naive(my_input, mapper, reducer):
map_results = map(mapper, my_input)
shuffler = defaultdict(list)
for key, value in map_results:
shuffler[key].append(value)
return map(reducer, shuffler.items())
This compact baseline maps each input item, collects values in a list per key with defaultdict, then applies the reducer to each group. section 3.2.3
The baseline makes the data flow clear, but it does not run mapper or reducer calls in parallel. Its returned map object is lazy: converting it to a list executes the reducers and exposes the results. It is a starting point for a faster engine, not the parallel implementation. section 3.2.3
To serve multiple clients, an asynchronous TCP frontend can submit jobs through a queue to a worker thread. That thread manages a multiprocessing pool for chunked MapReduce work and places completed results on another queue. The separation keeps request handling asynchronous while processes perform the computation. Queues synchronize communication between threads; transferring pickled objects between processes can add cost. section 3.5 section 3.5.1
