liveBook · Manning

Scraping

As explained in Learn AI Data Engineering

Web scraping is the automated or manual extraction of information from web pages, typically by processing their underlying markup.

Web scraping extracts information from web pages and turns presentation-oriented HTML into structured records. Unlike APIs, databases, or CSV files, pages rarely offer a stable schema: useful values may be nested in markup alongside navigation, ads, scripts, and formatting artifacts. The practical task is to locate the relevant elements, extract their contents, clean them, and map them to fields a pipeline can use. section 10.0 section 10.1 section 10.1.1

<div class="product-container">
  <div class="product-header">
    <h1 class="product-title">Men's Alpine Insulated Jacket</h1>
    <span class="brand-name">Mountain Gear Co.</span>
  </div>
  <div class="product-pricing">
    <span class="price-current">$189.99</span>
    <span class="price-original" style="text-decoration: line-through;">$249.99</span>
  </div>
  <div class="product-details">
    <p class="description">Stay warm in extreme conditions with our flagship
    insulated jacket. Features 800-fill down, waterproof shell, and
    adjustable hood.</p>
    <ul class="specs">
      <li>Weight: 1.2 lbs</li>
      <li>Material: Nylon ripstop</li>
      <li>Insulation: 800-fill goose down</li>
    </ul>
  </div>
  <!-- 200 more lines of navigation, scripts, and tracking pixels -->
</div>

This product-page fragment shows why visible fields are not ready-made records: title, brand, price, description, and specifications must be identified amid nested presentation markup. section 10.1.1

Prefer an API when one is available: APIs are designed for programmatic use, documented, and generally more stable than page layouts. Scraping can be appropriate when publicly available, permitted data is absent from an API, an API is incomplete, or multiple needed sources lack APIs. It is not a way around paywalls, paid plans, licenses, terms, rate limits, or applicable law; obtain access or permission when required. section 10.1.2

A practical scraping workflow moves from page inspection to repeatable pipeline work:

  • Choose a trustworthy product URL, preferably a manufacturer page when available, and identify fields such as name, price, description, images, weight, materials, and dimensions. section 10.4.3
  • Use browser developer tools and the selector tool to connect visible values to their HTML tags, classes, attributes, and surrounding structure. section 10.4.3
  • Fetch the URL in code; store raw responses, retry failures, log metadata, and batch-process pages for reproducibility and scale. section 10.5
  • Parse the retrieved markup into a navigable tree, then search tags and attributes, extract text, and clean formatting artifacts. section 10.1.1 section 10.5.1
  • Normalize extracted values into a consistent schema, while accounting for site-specific layouts, variants, sales pages, redesigns, and missing fields. section 10.5.1 section 11.1

Extracting outdoor-product records

An outdoor gear catalog may need product name, brand, price, description, weight, materials, image URL, and category from manufacturer pages. A page can visibly present these values while placing them in different nested elements across sites. A classic Python harness uses requests to retrieve each page and BeautifulSoup to navigate its HTML; conventional normalization then relies on selectors, regular expressions, and edge-case handling for each site. section 10.0 section 10.5.1 section 11.1

Fetching HTML is only the first dependency: without reliable retrieval, parsing, field extraction, and large-scale processing cannot proceed. The enduring mechanics are HTTP retrieval, DOM inspection, HTML parsing, selectors, and text extraction. The costly part at scale is normalization across inconsistent pages; later AI-assisted workflows can reduce manual URL-ranking and schema-mapping work, but require the same human-controlled attention to access rights and extraction quality. section 10.5 section 10.5.1 section 11.1

How Learn AI Data Engineering covers it

How deep it goes
Works through inspection, retrieval, HTML parsing, extraction, batching, retries, logging, and normalization, while deferring AI-assisted scaling details.
Examples
Uses outdoor-product pages to extract titles, prices, descriptions, images, weights, materials, and variants from nested HTML.
What it assumes
Assumes familiarity with programmatic data sources while teaching HTML inspection and classic requests-plus-BeautifulSoup scraping mechanics.
What you can do after
Operate/configure it as a reproducible Python pipeline that fetches, parses, extracts, logs, retries, and batches permitted pages.
How it connects
Connects HTML, CSS selectors, browser developer tools, HTTP requests, BeautifulSoup, APIs, data pipelines, and schema normalization.

Also explained in

Compare how 10 books explain scraping →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.