Imagine dividing a mountain of mail among many helpers, letting each helper process a pile, and then combining their results. MapReduce follows this pattern: map applies one operation independently to partitioned inputs, while reduce aggregates the intermediate results. section 7.8 section 13.10
A MapReduce computation usually proceeds in two phases:
- Map divides input data into chunks and transforms each chunk into intermediate key-value pairs. section 7.8 section 13.10
- Reduce groups values belonging to each unique key and combines them, producing a consolidated result. section 7.8 section 13.10
- Multiple processing resources can perform the map work in parallel, and multiple reduce tasks can divide the keys among themselves. section 7.8 section 13.10
Counting words
For text files, map splits the input into words and emits pairs such as ("the", 1), ("take", 1), and ("the", 1). Reduce groups the pairs by key and combines the values, producing ("the", 2). section 13.10
MapReduce resembles fork/join, but its map and reduce stages are more independent and can scale beyond one computer. Frameworks can distribute work across hundreds or thousands of machines, while developers provide the mapping and reduction logic. Spark follows a related model and can cache results across jobs instead of writing each job's data to disk. section 7.8
Visualizes map producing intermediate results before reduce combines them. section 7.8
