liveBook · Manning

Chapter 11: AI-powered web scraping

What you will learn in chapter 11 of Learn AI Data Engineering

This introduction explains why the manual RuckZone scraping approach does not scale and previews an incremental, stage-by-stage use of AI in the enrichment pipeline.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why the manual RuckZone scraping approach does not scale and previews an incremental, stage-by-stage use of AI in the enrichment pipeline.

Key terms: BeautifulSoup, CSS selectors, variant-specific logic, fallback strategies, RuckZone enrichment pipeline, AI opportunities, AI-assisted URL discovery, HTML cleaning, product extraction, token costs

11.1 Where we left off

Recaps the six-stage product enrichment pipeline and the manual scraping problems that motivate AI assistance.

Key terms: enrichment pipeline, Data Preparation, URL Discovery, HTML Fetching, Image Extraction, AI Extraction, selector treadmill, variant pages, inconsistent HTML, manual URL selection

11.2 Recognizing AI Opportunities in Data Pipelines

Introduces a test for spotting pipeline tasks suited to AI and names three categories of AI-generated data.

Key terms: deterministic code, unstructured content, structured fields, schema, extracted data, enriched data, synthetic data

11.2.1 Extracted Data

This section defines extracted data and introduces LLM extraction as a replacement for site-specific CSS selectors.

Key terms: extracted data, AI extraction, schema, prompt, CSS-selector-based scraping

11.2.2 Enriched Data

This section distinguishes enriched data from extracted data and explains why inferred fields need more careful validation.

Key terms: enriched data, extraction, external sources, judgment, enriched fields, validate

11.2.3 Synthetic Data

The section defines synthetic data, explains where it is useful and risky, and distinguishes it from the chapter’s focus on extracting and enriching real product data.

Key terms: synthetic data, extracted data, enriched data, production enrichment, ground truth, schema validation, load testing

11.3 Mapping AI to the Enrichment Pipeline

This section maps AI onto the six-stage web-scraping enrichment pipeline and identifies where it is worth using.

Key terms: enrichment pipeline, Stage 1, stages 2, 3, and 5, manual approaches, AI payoff

11.4 AI-Assisted URL Discovery

This section introduces a two-step way to replace slow, inconsistent manual product URL selection with search API candidates ranked by an LLM.

Key terms: product URLs, manually, inconsistent, search API, candidate URLs, LLM, rank

11.4.1 Finding Candidate URLs Programmatically

This section replaces manual product-page searching with a search API that returns candidate URLs for later AI ranking.

Key terms: SerpAPI, search API, candidate URLs, snippet, search position, SERPAPI_KEY, .env file, candidates list, ranking function

11.4.2 Ranking URLs with AI

This section uses an LLM to rank search-result URLs for product-data extraction, then checks the structured choice before using it downstream.

Key terms: candidate URLs, product-detail pages, manufacturer or brand pages, structured product data, URLRanking, Pydantic data class, system prompt, best_url, confidence, reasoning

11.5 Smarter HTML Cleaning

This section shows how to shrink product-page HTML with rules-based cleaning before sending it to an LLM for extraction.

Key terms: cleaned HTML, tokens, LLM extraction, BeautifulSoup, tag-level HTML cleaning, class-name patterns, rules-based, basic cleaning, aggressive cleaning, character counts

11.6 From Manual Selectors to AI Extraction

This section contrasts brittle, site-specific CSS selectors with AI extraction that uses a schema and prompt to specify desired fields.

Key terms: CSS selectors, AI extraction, schema, prompt

11.6.1 Defining What You Want

This section defines a Pydantic product-extraction schema that guides the model’s output while accommodating incomplete product pages.

Key terms: ProductExtraction, Pydantic model, target schema, product_name, brand_name, Field(description=...), response_format, structured output, implicit prompt engineering, price

11.6.2 Letting AI Do the Extraction

The section shows how an LLM can extract schema-shaped product data from cleaned HTML, and explains the prompt rules and tradeoffs that make the approach useful across sites.

Key terms: cleaned HTML, LLM, Pydantic schema, structured product data, explicit presence, null, sale price, original price, hallucinating fields, hallucinate

11.6.3 Manual vs. AI: A Side-by-Side Comparison

A side-by-side product-page comparison shows that AI extraction captures more information than a manual selector and can reuse one function across many sites.

Key terms: manual approach, AI approach, selectors, GR1, GR1 26L, extract_manual_goruck, site-specific extractors, AI function

11.7 Scaling Extraction Across Multiple Sites

The section introduces a batch workflow that repeats AI-assisted product extraction across several pages while recording results and continuing past failures.

Key terms: candidate URLs, cleaned HTML, extraction step, batch workflow, curated list, sample input, orchestration logic, companion notebook

11.7.1 One Prompt, Many Sites

11.7.2 Handling Failures and Partial Results

This section explains why product-page extraction can fail or return incomplete data, how to record the outcome, and what a production pipeline needs to handle unreliable results.

Key terms: status field, success, error, partial, requests.get(), retry logic, transient failures, logging, review queue, low-confidence results

11.8 Cost and Token Awareness

This section explains how to estimate LLM extraction costs and control them through HTML cleaning and task-appropriate model choices before running a large batch.

Key terms: input tokens, output tokens, tiktoken, token count, cost estimate, aggressive HTML cleaning, URL ranking, content triage, gpt-4o-mini, gpt-4o

11.9 Building Your AI Opportunity Checklist

This section turns the chapter’s examples into a checklist for deciding where AI belongs in a data pipeline, how to justify its cost, and how to handle errors.

Key terms: unstructured or semi-structured, smart intern test, site-specific selectors, per-record cost, regex, simple transformations, exact matching, high-volume, low-value decisions, schema checks, spot-sampling

11.10 Lab

This lab has readers run AI-assisted product extraction on a varied batch, then assess field coverage, cost, and how it compares with manual scraping.

11.11 Lab Answers

Where this is used later

Concepts explained in this chapter

← Chapter 10: Introduction to Web Scraping Chapter 12: AI-powered web scraping data pipelines in action →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.