An intermediate level divides a parallel workflow into smaller aggregation jobs. It receives grouped partial results from level 1, processes each group separately, and passes compact results to the final aggregation stage. section 13.2.1
Use the pattern when a final coordinator would otherwise become the bottleneck:
- Partition level-1 outputs into groups assigned to separate workers. section 13.2.1
- Have each intermediate worker aggregate the entries in its assigned lists. section 13.2.1
- Send the smaller intermediate results to the final aggregator. section 13.2.1
Parallel coin counting
Counting machines produce lists of coin totals. Instead of sending every list to one final worker, each family member receives a group, sums totals by denomination, and forwards a short result. The final worker handles aggregated values rather than hundreds of large lists. section 13.2.1
Figure 13.5 Revised coin counting process, using MapReduce and a little help. Now, each counting machine produces several lists, one for each member of the McDuck family at level 2. They, in turn, will have to check a hundred lists, but each with just a few entries, and sum up the values in each list by entry.
Compare the revised workflow: counting machines produce lists for level-2 workers, which sum short denomination lists before final aggregation. section 13.2.1
