liveBook · Manning

Chapter 10: Introduction to Web Scraping

What you will learn in chapter 10 of Learn AI Data Engineering

Introduces web scraping through a product-catalog project and explains why manual extraction comes before AI-powered enrichment.

After this chapter you will be able to

Section by section

Introduction

Introduces web scraping through a product-catalog project and explains why manual extraction comes before AI-powered enrichment.

Key terms: web scraping, HTML, structured records, RuckZone, product database, data contract, manual, AI-powered, extraction, generation

10.1 Why Web Scraping?

This section contrasts programmatically accessible data sources with web pages and explains why HTML makes data extraction less predictable.

Key terms: HTML, presentation format, data exchange format, REST APIs, JSON, nested `<div>` tags, schema, contract, navigation menus, advertisements

10.1.1 The Gap Between Display and Data

A product page may look like organized data in a browser, but scraping must turn its presentation-oriented HTML into usable fields.

Key terms: web scraping, HTML elements, presentation markup, text content, formatting artifacts

10.1.2 When Scraping Makes Sense

The section explains when web scraping is justified instead of an API and what permissions are required before collecting web data.

Key terms: API, web scraping, HTML layouts, RuckZone, publicly available for free, site’s terms, applicable law, paywall, licensed, SerpAPI

10.2 The Product Enrichment Challenge

Introduces RuckZone’s challenge of turning a starter product list into a searchable, analysis-ready catalog and frames it as a pipeline developed in later chapters.

Key terms: RuckZone, starter list of products, structured catalog, data contract, enrichment pipeline

10.2.1 From a Simple Product List to a Structured Catalog

The section defines RuckZone’s product-enrichment problem: turning a minimal product list into records useful for customers and downstream systems.

Key terms: Brand Name, Product Name, product enrichment, complete product record

10.2.2 The Enrichment Pipeline

This section lays out a six-stage pipeline for turning a product list into enriched catalog entries and identifies which stages this chapter tackles.

Key terms: Data Preparation, URL Discovery, HTML Fetching, Image Extraction, AI Extraction, Target Mapping, Stages 1 and 3, manual extraction, fragile at scale

10.3 Loading the Product Data

The section loads a product spreadsheet into a DataFrame and creates a consistent search key for each row so products can move through a repeatable enrichment pipeline.

Key terms: DataFrame, search_key, brand name, product name, search key, unit of work, URL discovery, enrichment pipeline, unique brands, category

10.4 Finding a Product URL Manually

This section uses one product to show the judgments involved in finding a scrapeable URL manually and why those judgments become a bottleneck at scale.

Key terms: URL discovery, search query, relevance, credibility, best source, stable URL, bottlenecks, ranking candidates, trade-offs

10.4.1 Search for the product page

This subsection starts manual product-page discovery by searching for a specific product and scanning the range of results before opening one.

Key terms: GORUCK GR1 26L, manufacturer site, retailers, review sites, forum posts

10.4.2 Evaluate the results and choose a source

This section explains how to choose a product page that supports extraction without mistaking a convenient source for an unquestionable authority.

Key terms: manufacturer page, product page, enrichment pipeline, extraction, primary sources, corroborate, key fields

10.4.3 Open the page and inspect what is actually available

This section shows how to inspect a product page for extractable data and HTML elements, then uses a manual exercise to reveal why URL discovery needs automation.

Key terms: Inspect, Inspect Element, developer tools, Elements panel, selector tool, HTML structure, product page URL, manufacturer page, HTML tag, CSS class

10.5 Web Scraping Fundamentals

This section frames reliable page retrieval as the first step that turns web scraping into a data engineering workflow, while noting that downloaded pages still need parsing and extraction.

Key terms: product URL, retrieve the page contents, fetch, data source, pipeline, raw responses, reproducibility, retry failures, log metadata, page contents

10.5.1 The Classic Scraping Harness: requests plus BeautifulSoup

Introduces requests and BeautifulSoup as the reliable scraping foundation, then explains why AI is used later to normalize inconsistent pages rather than relying on extensive site-specific rules.

Key terms: requests, BeautifulSoup, bs4, HTTP, HTML, navigable tree, consistent schema, site-specific rules, selectors, edge case handling

10.5.2 Fetching raw HTML with requests

This section explains what fetching a web page returns and why its raw HTML needs cleaning before extraction.

Key terms: HTTP response, status code, headers, 200 status code, html_content, raw HTML, page source, navigation menus, scripts, stylesheets

10.5.3 Parsing and cleaning HTML with BeautifulSoup

BeautifulSoup turns fetched HTML into cleaner text, but that text still needs site-dependent extraction and normalization.

Key terms: BeautifulSoup, tree-like structure, scripts, styles, navigation, footers, product information, normalizing, extracting, classic Python stack

10.6 Examining Real HTML Structure

This section shows how to inspect parsed product-page HTML, extract a title and price with selectors and fallbacks, and recognize why site-specific extraction becomes difficult to scale.

Key terms: Inspect, HTML structure, soup, selectors, select_one, CSS selector, price-item--sale, price-item--regular, get_text(" ", strip=True), scraping harness

10.7 Manual Extraction Across Multiple Sites

The section tests title-and-price extraction across two product sites to show why manual scraping needs site-specific rules and becomes hard to scale.

Key terms: site-specific selectors, structured-data parsing, fallback strategy, product URLs, requests.Session(), requests.get(), browser-like headers, 403 errors, manual extraction, requests plus BeautifulSoup

10.8 Lab

This lab builds a reusable scraper for 15 product URLs, handles both ordinary and variant-specific prices, and reports how many products were scraped successfully.

Where this is used later

Concepts explained in this chapter

← Chapter 9: Data Cleaning and Transformation Pipelines in Practice Chapter 11: AI-powered web scraping →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.