liveBook · Manning

MapReduce

As explained in Retrieval Augmented Generation, The Foundational Ideas

MapReduce is a programming model and distributed computation pattern that independently maps input records and then reduces intermediate results into an aggregate output.

MapReduce divides distributed computation into independent mapping work and reduction into an aggregate result. In GraphRAG, it organizes global question answering over precomputed community summaries rather than raw documents. section 12.2 section 12.2.2

A GraphRAG MapReduce query follows this sequence:

  • Select a hierarchy level whose community summaries balance detail, breadth, and token cost. section 12.2.2
  • Shuffle summaries to reduce positional anchoring, then pack them into context-window-sized batches. section 12.2.2
  • Map each batch into a helpfulness score and a query-specific partial answer; discard zero-helpfulness results. section 12.2.2
  • Reduce the surviving answers by ranking them, filling the final context window, and synthesizing one response. section 12.2.2

Recurring outage causes

For ten years of SaaS incident reports, naive retrieval may return the loudest individual outages. GraphRAG instead summarizes communities such as authentication, data, and deployment, then maps the global question across those summaries and reduces the useful partial answers into a cross-cutting explanation. section 12.2 section 12.2.2

Figure 12.3 The summarization pyramid. The four phases of GraphRAG (extraction, community detection, hierarchical summarization, and Map-Reduce search) build and query this structure.

Use the summarization pyramid to trace how raw chunks become entity communities, hierarchical summaries, and a MapReduce query path. section 12.2

MapReduce shifts substantial work into indexing: understanding is computed once and queried repeatedly. This supports large corpora, but requires heavier indexing and careful choices about summary level, context limits, and helpfulness filtering. section 12.2 section 12.2.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains batching, scoring, filtering, ranking, and synthesis, while deferring implementation beyond the query-time procedure.
Examples
Uses SaaS incident reports, audit corpora, community summaries, batch scores, and a summarization pyramid.
What you can do after
Apply it via GraphRAG to answer broad corpus questions from ranked, precomputed community summaries.
How it connects
Connects MapReduce to GraphRAG, hierarchical summarization, community detection, retrieval-augmented generation, and context windows.

Also explained in

Compare how 8 books explain MapReduce →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.