Reduce task

As explained in Grokking Concurrency

A reduce phase is the stage of a map/reduce computation that combines intermediate values associated with keys into final results.

Imagine several workers counting words in separate stacks of pages. A reduce task gathers their intermediate key-value pairs and combines the values belonging to each unique key, turning repeated partial results into final totals. section 13.10

A reduce phase follows a simple handoff:

• Receive the map output as key-value pairs. section 13.10
• Group values associated with each unique key. section 13.14
• Call the reduction function for each key and write the final output. section 13.14

Counting words

If mapping produces [("the", 1), ("take", 1), ("the", 1)], reduction combines the two values for "the" and returns [("the", 2)]. A larger implementation can assign word lists to reduce tasks, while the scheduler starts reduction after all map tasks finish. section 13.10 section 13.13

In a concurrent pipeline, workers process intermediate data, invoke reducefn once per unique key, and write results to a file. The scheduler uses distinct mapping, reducing, and finished states; the implementation uses one workflow for its single reduce task. section 13.14

Use the program overview to trace how map outputs flow through scheduling into the reduce phase. section 13.14

How Grokking Concurrency covers it

How deep it goes

Defines aggregation, demonstrates it with word counts, and implements worker, scheduler, and output-file behavior.

Examples

Uses a three-pair word-count example, distributed word lists, and a concurrent text-processing implementation.

What it assumes

Assumes map output as key-value pairs and teaches grouping and aggregation before showing distributed scheduling.

What you can do after

Build it from scratch as a worker-based pipeline that groups intermediate values and writes one final result file.

How it connects

Connects map tasks, key-value pairs, combine tasks, workers, and central scheduling.

liveBookmanning.com

Read Grokking Concurrency on liveBook Get the book at manning.com

Related concepts

• name-value pairuses · A name-value pair is a data association in which an identifying name or key corresponds to a value.
• agglomerationused by · Agglomeration is the combination of smaller computational tasks into larger units to reduce coordination and communication costs.
• MapReducepart of · MapReduce is a computational pattern that applies map operations to input elements or partitions and combines the intermediate results through reduction or aggregation.
• taskkind of · A task is a defined unit of work, computation, or action directed toward a result or objective.

This explainer was generated by Manning's knowledge graph from the text of Grokking Concurrency. The section links point to the parts of the book it draws on.