MapReduce processes data in two stages: independently transform smaller chunks, then combine the partial results into one final result. For large-language-model summarization, it keeps each prompt within the model’s context window. section 3.1.3 section 3.1
A document summarization pipeline typically works as follows:
- Split the document into token-sized chunks with a tokenizer such as TokenTextSplitter. section 3.1
- Map the same summarization chain over every chunk, independently and in parallel. section 3.1.3
- Reduce the intermediate summaries into a final summary. section 3.1.3
Summarizing a large report
A 100-page report is split into chunks, summarized separately, and then condensed from those partial summaries. This supports parallel processing and lower token costs, but may lose contextual connections between sections. section 3.4
A LangChain implementation composes a prompt, language model, and StrOutputParser with the pipe operator, then uses RunnableParallel to apply the map chain across chunks. Across multiple documents, each Document becomes an intermediate summary before reduction. section 3.1.3 section 3.2
Figure 3.1 Summarizing a document bigger than the LLM’s context window involves splitting the document into smaller chunks, summarizing each chunk, and then summarizing the combined chunk summaries.
Shows the complete flow from splitting an oversized document through chunk summaries to the final combined summary. section 3.1
Choose MapReduce when the input exceeds the context window and parallel speed matters more than preserving every contextual connection. Use refine instead when each document’s essence must be retained; refine processes chunks sequentially and updates a running summary. section 3.2 section 3.3
