Scraping is the programmatic extraction of text from web pages. In a research summarization engine, it turns result URLs into source material that an LLM can summarize, preserve with its URL, and combine into a report. section 4.1 section 4.7.3
A practical scraping stage passes each search result through a small processing chain:
- Receive a dictionary containing the result URL, search query, and user question. section 4.7.3
- Call a web_scrape function with the URL and retain at most 10,000 characters of returned text. section 4.7.3
- Send the extracted text to a summary prompt while passing the source URL and question alongside the result. section 4.7.3
- Map the chain across multiple URLs in parallel, then join the individual summaries into research input. section 4.7.3
Astorga research query
For a question about things to see in Astorga, search results provide URLs such as travel and attraction pages. Scraping one page produces text that can be summarized into findings about attractions, history, cuisine, and natural beauty, with the source URL included. section 4.6.4 section 4.7.3
Listing 4.10 Search Result Text and Summary chain
from llm_models import get_llm
from web_scraping import web_scrape
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnableLambda, RunnableParallel
from prompts import (
SUMMARY_PROMPT_TEMPLATE
)
RESULT_TEXT_MAX_CHARACTERS = 10000
search_result_text_and_summary_chain = (
RunnableLambda(lambda x:
{
'search_result_text':
web_scrape(url=x['result_url'])[
:RESULT_TEXT_MAX_CHARACTERS],
'result_url': x['result_url'],
'search_query': x['search_query'],
'user_question': x['user_question']
}
)
| RunnableParallel (
{
'text_summary': SUMMARY_PROMPT_TEMPLATE
| get_llm() | StrOutputParser(),
'result_url': lambda x: x['result_url'],
'user_question': lambda x: x['user_question']
}
)
| RunnableLambda(lambda x:
{
'summary':
f"Source Url: {x['result_url']}\nSummary:
{x['text_summary']}",
'user_question': x['user_question']
}
)
)
Inspect the chain implementation showing URL scraping, text truncation, metadata preservation, and parallel summarization. section 4.7.3
