4 Building a research summarization engine

Building on the content summarization techniques from chapter 3, this chapter guides you through creating a research summarization engine. This LLM application will process user queries, perform web searches, and compile a comprehensive summary of the findings. We’ll develop this project step-by-step, starting with the basics and gradually increasing in complexity. Along the way, you’ll deepen your knowledge of LangChain as I introduce creating LLM chains with the LangChain Expression Language (LCEL).

4.1 Overview of a research summarization engine

Imagine you’re researching various topics, such as a specific NBA player, a tourist destination, or whether to invest in a stock. Manually, you’d perform a web search, sift through results, read related pages, take notes, and compile a summary. A modern approach is to let an LLM handle this work. You could copy text from each web page, paste it into a ChatGPT prompt for summarization, and repeat for multiple pages. Then, combine these summaries into a final prompt for a consolidated summary (see figure 4.1).

4.2 Setting up the project

4.3 Implementing the core functionality

4.3.1 Implementing web searching

4.3.2 Implementing web scraping

4.3.3 Instantiating the LLM client

4.3.4 JSON to Python object converter

4.4 Enhancing the architecture with query rewriting

4.5 Prompt engineering

4.5.1 Crafting web search prompts

4.5.2 Crafting summarization prompts

4.5.3 Research report prompt

4.6 Initial implementation

4.6.1 Importing functions and prompt templates

4.6.2 Setting constants and input variables

4.6.3 Instantiating the LLM client

4.6.5 Scraping the web results