Chapter 10: Introduction to Web Scraping
Introduces web scraping through a product-catalog project and explains why manual extraction comes before AI-powered enrichment.
After this chapter you will be able to
- Web pages as data sources. Explain why data on web pages requires a different approach from the structured API, database, and CSV data used earlier, and describe web scraping as extracting information from HTML into structured records.
- RuckZone product catalog challenge. Describe RuckZone's aim to build a comprehensive rucking-product database and the need to gather specific product attributes from pages across the web into records that meet a strict data contract.
- Manual scraping before AI. Explain that this chapter tackles scraping manually, without AI, to reveal the work and pain points, while later chapters bring AI into extraction, generation, and enrichment at scale.
- HTML as presentation format. Distinguish web pages from REST APIs, databases, and CSV files by explaining that HTML is designed to render content for people rather than provide a structured data exchange format.
- Unpredictable page data structure. Explain how desired data can be buried in nested HTML elements, spread across a page, or mixed with unrelated content, and why extraction cannot rely on a guaranteed schema or stable page structure.
- Display-to-data gap. Explain why a visually clear product page does not provide ready-made structured fields, and describe how scraping finds the relevant HTML elements, extracts their text, and cleans formatting artifacts.
- API versus scraping choice. Explain why an API is the first choice and when scraping becomes practical, using RuckZone’s product data as an example.
- Permission and licensing boundaries. State the conditions under which scraping is appropriate and explain why third-party search tooling does not grant rights to retrieved content.
- Product enrichment challenge. Explain that RuckZone has a starter list of rucking products and wants to enrich it into a searchable, analysis-ready structured catalog that meets a strict data contract.
- Product list enrichment gap. Explain that RuckZone starts with a spreadsheet containing only brand name and product name for each product, but needs richer product records with customer-relevant fields that downstream systems can validate.
- Six-stage product enrichment pipeline. Trace the six stages in order: prepare product search keys, discover and rank candidate URLs, fetch and clean HTML, extract image URLs, use an LLM to extract product attributes, and map and validate the results against the final schema.
- Product search key creation. Describe loading the product spreadsheet into a DataFrame and creating a search_key for each row by combining its brand name and product name into a phrase suitable for searching for a product page.
- Consistent inputs for enrichment. Explain why preparing a repeatable search key makes each product row a unit of work and helps later URL discovery, scraping, and matching extracted information back to the original list.
- Manual URL discovery decisions. Describe how a product name becomes a search query, how search results are judged for relevance and credibility, and how a source and stable URL are chosen for later scraping.
- URL discovery scaling bottleneck. Explain why finding a URL manually for one product is useful practice but becomes inconsistent and exhausting across hundreds of products, and how AI could later help rank candidates and explain trade-offs.
- Product search result scan. Describe searching for “GORUCK GR1 26L” in a browser and scanning the results before choosing a page, rather than automatically clicking the first result. Note that results may include the manufacturer, retailers, reviews or blogs, and forums.
- Product page source selection. Explain why the enrichment pipeline generally prefers a manufacturer’s product page and why the chosen URL matters for later extraction. Distinguish a product page from category pages, search results, or incomplete reseller listings.
- Source accuracy and corroboration. Explain why even a well-structured manufacturer page should not be treated as unquestioned truth. Describe when to favor primary sources and check key fields against more than one source before adding them to the dataset.
- Product page inspection. Describe how to examine a product page for fields that can be reliably extracted and use the browser’s Inspect tool to locate the HTML behind its title or price.
- Manual product page exercise. Apply the manual process to a different spreadsheet product: search by brand and product name, choose a suitable page, inspect its title and price, and record the URL, the reason for choosing it, and the price’s likely HTML tag or CSS class.
- Reliable page retrieval. Explain that, once a product URL is known, scraping first requires retrieving the page in code; without a reliable fetch, later parsing, field extraction, and scaling cannot work.
- Classic scraping library roles. Explain that requests fetches a web page from a URL as an HTTP response, while BeautifulSoup parses its raw HTML into a navigable tree for finding tags, attributes, and text.
- Normalization beyond page fetching. Explain that the difficult part of scraping across many sites is turning inconsistent product pages into a consistent schema, such as price, weight, description, category, and image URL. Contrast traditional site-specific selectors and edge-case rules with the book’s approach: keep requests and BeautifulSoup for fetching and parsing, then use AI for the messy normalization work in later chapters.
- HTTP response and raw HTML. Describe the response received when a page is fetched: it includes a status code, headers, and page content; a 200 response provides HTML that is held as a string in html_content.
- BeautifulSoup text cleaning. Describe how BeautifulSoup parses HTML into a tree so unwanted sections can be removed, then combines the remaining text into one normalized string.
- Limits of cleaned page text. Explain why cleaned HTML text is not yet structured product data: other page content remains mixed in, and content order and formatting vary by site. Distinguish straightforward fetching and parsing from the harder work of normalization and extraction.
- Evidence-based HTML inspection. Explain how inspecting a page in the browser relates to examining its parsed BeautifulSoup object, and why you should locate the HTML containing a field before writing a selector.
- Title and price extraction. Describe how the example uses CSS selectors to find product fields, preferring a sale price and falling back to a regular price. Explain how it turns a matched element into clean text and handles a missing match.
- Manual scraping at scale. Explain why extracting fields manually becomes repetitive across many products and domains, and how that motivates the later use of AI with the same scraping harness.
- Two-site extraction approaches. Describe how extracting titles and prices differs between the GORUCK and Osprey product pages: the example uses site-specific selectors for GORUCK and structured-data parsing with a fallback for Osprey. State that the correct product URLs are assumed rather than discovered here.
- Manual scraping scale limits. Explain why manual extraction becomes difficult across domains: HTML layouts require per-site rules, requests may face access controls, throttling, timeouts, or JavaScript-heavy pages, and changing layouts create maintenance work. Describe the later direction as retaining the requests-and-BeautifulSoup harness while using AI to interpret page content and normalize results into a consistent schema.
Section by section
Introduction
Introduces web scraping through a product-catalog project and explains why manual extraction comes before AI-powered enrichment.
- Web pages as data sources: Explain why data on web pages requires a different approach from the structured API, database, and CSV data used earlier, and describe web scraping as extracting information from HTML into structured records.
Why it matters: This distinction explains why the chapter needs a new extraction workflow. - RuckZone product catalog challenge: Describe RuckZone's aim to build a comprehensive rucking-product database and the need to gather specific product attributes from pages across the web into records that meet a strict data contract.
Why it matters: The catalog provides the practical reason for scraping and defines what the extracted data must become. - Manual scraping before AI: Explain that this chapter tackles scraping manually, without AI, to reveal the work and pain points, while later chapters bring AI into extraction, generation, and enrichment at scale.
Why it matters: The manual baseline makes the later AI-assisted workflow easier to evaluate.
Key terms: web scraping, HTML, structured records, RuckZone, product database, data contract, manual, AI-powered, extraction, generation
10.1 Why Web Scraping?
This section contrasts programmatically accessible data sources with web pages and explains why HTML makes data extraction less predictable.
- HTML as presentation format: Distinguish web pages from REST APIs, databases, and CSV files by explaining that HTML is designed to render content for people rather than provide a structured data exchange format.
Why it matters: This distinction explains why extracting data from a page takes different work from reading an API response or CSV file. - Unpredictable page data structure: Explain how desired data can be buried in nested HTML elements, spread across a page, or mixed with unrelated content, and why extraction cannot rely on a guaranteed schema or stable page structure.
Why it matters: Recognizing these obstacles helps you understand why scraping logic needs to account for variation in real pages.
Key terms: HTML, presentation format, data exchange format, REST APIs, JSON, nested `<div>` tags, schema, contract, navigation menus, advertisements
10.1.1 The Gap Between Display and Data
A product page may look like organized data in a browser, but scraping must turn its presentation-oriented HTML into usable fields.
- Display-to-data gap: Explain why a visually clear product page does not provide ready-made structured fields, and describe how scraping finds the relevant HTML elements, extracts their text, and cleans formatting artifacts.
Why it matters: This distinction explains why product information needs an extraction and cleaning step before it can be used as data.
Key terms: web scraping, HTML elements, presentation markup, text content, formatting artifacts
10.1.2 When Scraping Makes Sense
The section explains when web scraping is justified instead of an API and what permissions are required before collecting web data.
- API versus scraping choice: Explain why an API is the first choice and when scraping becomes practical, using RuckZone’s product data as an example.
Why it matters: Choosing an access method before building an enrichment pipeline avoids unnecessary scraping. - Permission and licensing boundaries: State the conditions under which scraping is appropriate and explain why third-party search tooling does not grant rights to retrieved content.
Why it matters: A technically accessible page is not necessarily a permitted source for a data pipeline.
Key terms: API, web scraping, HTML layouts, RuckZone, publicly available for free, site’s terms, applicable law, paywall, licensed, SerpAPI
10.2 The Product Enrichment Challenge
Introduces RuckZone’s challenge of turning a starter product list into a searchable, analysis-ready catalog and frames it as a pipeline developed in later chapters.
- Product enrichment challenge: Explain that RuckZone has a starter list of rucking products and wants to enrich it into a searchable, analysis-ready structured catalog that meets a strict data contract.
Why it matters: This target defines what the enrichment work is meant to produce.
Key terms: RuckZone, starter list of products, structured catalog, data contract, enrichment pipeline
10.2.1 From a Simple Product List to a Structured Catalog
The section defines RuckZone’s product-enrichment problem: turning a minimal product list into records useful for customers and downstream systems.
- Product list enrichment gap: Explain that RuckZone starts with a spreadsheet containing only brand name and product name for each product, but needs richer product records with customer-relevant fields that downstream systems can validate.
Why it matters: This gap defines the problem the enrichment pipeline is meant to solve.
Key terms: Brand Name, Product Name, product enrichment, complete product record
10.2.2 The Enrichment Pipeline
This section lays out a six-stage pipeline for turning a product list into enriched catalog entries and identifies which stages this chapter tackles.
- Six-stage product enrichment pipeline: Trace the six stages in order: prepare product search keys, discover and rank candidate URLs, fetch and clean HTML, extract image URLs, use an LLM to extract product attributes, and map and validate the results against the final schema.
Why it matters: The stages show how raw product records become structured, checked catalog entries. - Chapter focus and manual limits: Distinguish the chapter’s focus on preparing product data and fetching HTML from the full six-stage pipeline, and explain that manual extraction is attempted to show why it becomes fragile at scale.
Why it matters: This keeps the chapter’s hands-on work distinct from the complete enrichment process.
Key terms: Data Preparation, URL Discovery, HTML Fetching, Image Extraction, AI Extraction, Target Mapping, Stages 1 and 3, manual extraction, fragile at scale
10.3 Loading the Product Data
The section loads a product spreadsheet into a DataFrame and creates a consistent search key for each row so products can move through a repeatable enrichment pipeline.
- Product search key creation: Describe loading the product spreadsheet into a DataFrame and creating a search_key for each row by combining its brand name and product name into a phrase suitable for searching for a product page.
Why it matters: A consistent search phrase makes it possible to automate the searches you would otherwise perform product by product. - Consistent inputs for enrichment: Explain why preparing a repeatable search key makes each product row a unit of work and helps later URL discovery, scraping, and matching extracted information back to the original list.
Why it matters: Without consistent inputs, later steps become harder to automate and their results harder to evaluate. - Product list inspection and refinement: Describe checking the DataFrame for its product count, unique-brand count, and brand with the most products, and suggest an additional search-key term such as a product category to make future searches more consistent.
Why it matters: Inspecting the inputs and refining queries can make subsequent searches easier to manage.
Key terms: DataFrame, search_key, brand name, product name, search key, unit of work, URL discovery, enrichment pipeline, unique brands, category
10.4 Finding a Product URL Manually
This section uses one product to show the judgments involved in finding a scrapeable URL manually and why those judgments become a bottleneck at scale.
- Manual URL discovery decisions: Describe how a product name becomes a search query, how search results are judged for relevance and credibility, and how a source and stable URL are chosen for later scraping.
Why it matters: A usable product URL is the result of several judgments, not just a search. - URL discovery scaling bottleneck: Explain why finding a URL manually for one product is useful practice but becomes inconsistent and exhausting across hundreds of products, and how AI could later help rank candidates and explain trade-offs.
Why it matters: The manual walkthrough reveals which judgment-heavy steps are worth assisting when the dataset grows.
Key terms: URL discovery, search query, relevance, credibility, best source, stable URL, bottlenecks, ranking candidates, trade-offs
10.4.1 Search for the product page
This subsection starts manual product-page discovery by searching for a specific product and scanning the range of results before opening one.
- Product search result scan: Describe searching for “GORUCK GR1 26L” in a browser and scanning the results before choosing a page, rather than automatically clicking the first result. Note that results may include the manufacturer, retailers, reviews or blogs, and forums.
Why it matters: Scanning first gives you a choice of sources to evaluate for product information.
Key terms: GORUCK GR1 26L, manufacturer site, retailers, review sites, forum posts
10.4.2 Evaluate the results and choose a source
This section explains how to choose a product page that supports extraction without mistaking a convenient source for an unquestionable authority.
- Product page source selection: Explain why the enrichment pipeline generally prefers a manufacturer’s product page and why the chosen URL matters for later extraction. Distinguish a product page from category pages, search results, or incomplete reseller listings.
Why it matters: A suitable starting page makes downstream product-field extraction easier. - Source accuracy and corroboration: Explain why even a well-structured manufacturer page should not be treated as unquestioned truth. Describe when to favor primary sources and check key fields against more than one source before adding them to the dataset.
Why it matters: Extracting a field successfully does not establish that the field is accurate.
Key terms: manufacturer page, product page, enrichment pipeline, extraction, primary sources, corroborate, key fields
10.4.3 Open the page and inspect what is actually available
This section shows how to inspect a product page for extractable data and HTML elements, then uses a manual exercise to reveal why URL discovery needs automation.
- Product page inspection: Describe how to examine a product page for fields that can be reliably extracted and use the browser’s Inspect tool to locate the HTML behind its title or price.
Why it matters: Scraping code works with the page’s HTML, not just what a shopper sees on screen. - Manual product page exercise: Apply the manual process to a different spreadsheet product: search by brand and product name, choose a suitable page, inspect its title and price, and record the URL, the reason for choosing it, and the price’s likely HTML tag or CSS class.
Why it matters: Doing the steps yourself makes both the extraction task and the limits of manual discovery concrete. - Limits of manual discovery: Explain why manually finding and inspecting product pages does not scale reliably, and how later AI-assisted ranking and filtering is intended to help with URL selection.
Why it matters: The manual effort shows which repetitive judgments automation needs to reduce.
Key terms: Inspect, Inspect Element, developer tools, Elements panel, selector tool, HTML structure, product page URL, manufacturer page, HTML tag, CSS class
10.5 Web Scraping Fundamentals
This section frames reliable page retrieval as the first step that turns web scraping into a data engineering workflow, while noting that downloaded pages still need parsing and extraction.
- Reliable page retrieval: Explain that, once a product URL is known, scraping first requires retrieving the page in code; without a reliable fetch, later parsing, field extraction, and scaling cannot work.
Why it matters: Every later scraping step depends on getting the page content reliably. - Scraping as a pipeline workflow: Describe how programmatic fetching lets a website be handled like a pipeline data source, with raw responses stored for reproducibility, failed requests retried, metadata logged, and pages processed in batches.
Why it matters: These practices make scraping repeatable and manageable beyond a single browser session. - Human-facing page structure: Distinguish downloading a page from extracting structured product data: the page is built for human viewing, so parsing and field extraction remain after the fetch.
Why it matters: A successful download supplies the input for extraction, not the finished catalog data.
Key terms: product URL, retrieve the page contents, fetch, data source, pipeline, raw responses, reproducibility, retry failures, log metadata, page contents
10.5.1 The Classic Scraping Harness: requests plus BeautifulSoup
Introduces requests and BeautifulSoup as the reliable scraping foundation, then explains why AI is used later to normalize inconsistent pages rather than relying on extensive site-specific rules.
- Classic scraping library roles: Explain that requests fetches a web page from a URL as an HTTP response, while BeautifulSoup parses its raw HTML into a navigable tree for finding tags, attributes, and text.
Why it matters: These two steps provide the page content that any later extraction method needs. - Normalization beyond page fetching: Explain that the difficult part of scraping across many sites is turning inconsistent product pages into a consistent schema, such as price, weight, description, category, and image URL. Contrast traditional site-specific selectors and edge-case rules with the book’s approach: keep requests and BeautifulSoup for fetching and parsing, then use AI for the messy normalization work in later chapters.
Why it matters: It explains why a reliable scraping harness remains useful even when AI takes on part of extraction.
Key terms: requests, BeautifulSoup, bs4, HTTP, HTML, navigable tree, consistent schema, site-specific rules, selectors, edge case handling
10.5.2 Fetching raw HTML with requests
This section explains what fetching a web page returns and why its raw HTML needs cleaning before extraction.
- HTTP response and raw HTML: Describe the response received when a page is fetched: it includes a status code, headers, and page content; a 200 response provides HTML that is held as a string in html_content.
Why it matters: Checking the response and retaining its HTML gives you the input for the next extraction step. - Need to clean page source: Explain why the fetched HTML is not yet extraction-ready: it resembles browser page source and can contain navigation, scripts, stylesheets, analytics tags, and cookie banners, so the next step is to clean it.
Why it matters: Recognizing the noise in raw HTML explains why fetching and extracting are separate steps.
Key terms: HTTP response, status code, headers, 200 status code, html_content, raw HTML, page source, navigation menus, scripts, stylesheets
10.5.3 Parsing and cleaning HTML with BeautifulSoup
BeautifulSoup turns fetched HTML into cleaner text, but that text still needs site-dependent extraction and normalization.
- BeautifulSoup text cleaning: Describe how BeautifulSoup parses HTML into a tree so unwanted sections can be removed, then combines the remaining text into one normalized string.
Why it matters: This produces usable text from a page before product details are extracted. - Limits of cleaned page text: Explain why cleaned HTML text is not yet structured product data: other page content remains mixed in, and content order and formatting vary by site. Distinguish straightforward fetching and parsing from the harder work of normalization and extraction.
Why it matters: It explains why a single manual scraper is fragile across sites.
Key terms: BeautifulSoup, tree-like structure, scripts, styles, navigation, footers, product information, normalizing, extracting, classic Python stack
10.6 Examining Real HTML Structure
This section shows how to inspect parsed product-page HTML, extract a title and price with selectors and fallbacks, and recognize why site-specific extraction becomes difficult to scale.
- Evidence-based HTML inspection: Explain how inspecting a page in the browser relates to examining its parsed BeautifulSoup object, and why you should locate the HTML containing a field before writing a selector.
Why it matters: Inspecting the actual HTML helps you choose selectors based on evidence rather than guesses. - Title and price extraction: Describe how the example uses CSS selectors to find product fields, preferring a sale price and falling back to a regular price. Explain how it turns a matched element into clean text and handles a missing match.
Why it matters: Fallbacks and missing-element checks make extraction more useful when a product page does not have exactly one expected price element. - Manual scraping at scale: Explain why extracting fields manually becomes repetitive across many products and domains, and how that motivates the later use of AI with the same scraping harness.
Why it matters: The repeated site-specific work identifies where AI-assisted extraction could help a data pipeline scale.
Key terms: Inspect, HTML structure, soup, selectors, select_one, CSS selector, price-item--sale, price-item--regular, get_text(" ", strip=True), scraping harness
10.7 Manual Extraction Across Multiple Sites
The section tests title-and-price extraction across two product sites to show why manual scraping needs site-specific rules and becomes hard to scale.
- Two-site extraction approaches: Describe how extracting titles and prices differs between the GORUCK and Osprey product pages: the example uses site-specific selectors for GORUCK and structured-data parsing with a fallback for Osprey. State that the correct product URLs are assumed rather than discovered here.
Why it matters: The two pages show why one set of extraction rules cannot be assumed to work across sites. - Shared scraping session: Explain why the example uses requests.Session() instead of separate requests.get() calls: it reuses connections and provides one place to set browser-like headers across requests. Explain that those headers may avoid simple 403 blocks but do not guarantee access.
Why it matters: A shared session makes multi-page requests more consistent without solving every access problem. - Manual scraping scale limits: Explain why manual extraction becomes difficult across domains: HTML layouts require per-site rules, requests may face access controls, throttling, timeouts, or JavaScript-heavy pages, and changing layouts create maintenance work. Describe the later direction as retaining the requests-and-BeautifulSoup harness while using AI to interpret page content and normalize results into a consistent schema.
Why it matters: These limits motivate the move from hand-maintained extraction rules to AI-assisted extraction.
Key terms: site-specific selectors, structured-data parsing, fallback strategy, product URLs, requests.Session(), requests.get(), browser-like headers, 403 errors, manual extraction, requests plus BeautifulSoup
10.8 Lab
This lab builds a reusable scraper for 15 product URLs, handles both ordinary and variant-specific prices, and reports how many products were scraped successfully.
Where this is used later
- Designing the data contract (chapter 12) · Formalizes the data contract for the pipeline.
- Where we left off (chapter 11) · Revisits the manual scraping work as the starting point for AI assistance.
- Manual vs. AI: A Side-by-Side Comparison (chapter 11) · Compares manual extraction with AI extraction.
- From Manual Selectors to AI Extraction (chapter 11) · Revisits manual selectors and AI extraction in response to the difficulty of locating page data.
- Mapping AI to the Enrichment Pipeline (chapter 11) · Maps AI opportunities to stages of the enrichment pipeline.
- The four agents (chapter 12) · Implements pipeline responsibilities as agents.
- Finding Candidate URLs Programmatically (chapter 11) · Finds candidate product URLs programmatically.
- Orchestrating the pipeline (chapter 12) · Orchestrates the enrichment steps as a pipeline.
Concepts explained in this chapter
- Beautiful SoupBeautifulSoup is a Python library for parsing HTML and XML documents into navigable structures and extracting or modifying their contents.
- scrapingWeb scraping is the automated or manual extraction of information from web pages, typically by processing their underlying markup.
- HTMLHTML is a markup language for structuring content in web documents, including the source text retrieved from a web server.
- fallbackA fallback strategy is an approach that specifies an alternative method when the preferred method fails or cannot produce a result.
- Session classA Requests session is an object that persists settings and can reuse connections across multiple HTTP requests.
- data frameA pandas DataFrame is a two-dimensional labeled tabular data structure in the pandas library.
- requestsRequests is a Python library for sending HTTP requests and handling their responses.
- manual extractionManual extraction is the creation and maintenance of explicit rules for retrieving selected fields from source pages.
- site-specific selectorA site-specific selector is a webpage selection rule tailored to the HTML structure of a particular website.
- url discoveryURL discovery is the process of finding and selecting a web address for a particular product or other target item.
- schemaA database schema is the formal organization of a database's tables, fields, relationships, constraints, and related definitions.
- structured extractionAI-powered extraction is the use of an artificial intelligence model to identify and retrieve requested fields from source content based on field descriptions rather than fixed location rules.
- data contractA data contract is a formal specification of the structure, fields, types, and validity requirements that data exchanged between processing components must satisfy.
- JSONJSON is a text-based data interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
- AI-assisted ranking and filteringAI ranking is the evaluation and ordering of candidate items by an artificial intelligence model according to specified criteria.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
