MapReduce is a multistage processing pattern for handling large documents. It splits input into smaller chunks, processes each chunk independently, and combines the intermediate results into one higher-level result. section 7.1.2
A document-summarization workflow typically proceeds as follows:
- Split the document into manageable chunks, preserving some overlap so context survives chunk boundaries. section 7.1.3
- Map each chunk independently through an LLM summarization step, producing intermediate summaries. section 7.1.2
- Reduce the intermediate summaries by passing them through another summarization step that produces one combined overview. section 7.1.2
LangChain.js summarization chain
After loading documents and creating chunks, configure the chain with type "map_reduce" and invoke it with input_documents. Setting verbose to true enables logging while the workflow runs. section 7.1.3
Figure 7.2 The MapReduce summarization pattern, behind the scenes. The input document is divided into smaller chunks. Each chunk is then processed independently by the map function, generating intermediate summary pairs. These pairs are subsequently organized and are grouped together. The reduce step is then applied to these grouped pairs, aggregating the values to produce a final summary.
Trace the pattern visually from document chunks through independent map results, grouping, and the final reduce aggregation. section 7.1.2
MapReduce supports parallel processing and avoids sending an entire extensive document to the model in one call. Alternative summarization strategies include stuffing, which uses one call for the whole document, and refine, which updates a summary iteratively for each chunk. section 7.1.2 section 7.3
