liveBook · Manning

Chapter 12: AI-powered web scraping data pipelines in action

What you will learn in chapter 12 of Learn AI Data Engineering

The chapter turns single-product web-scraping snippets into a batch pipeline built from single-purpose agents, with the supporting machinery needed to run it unattended.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter turns single-product web-scraping snippets into a batch pipeline built from single-purpose agents, with the supporting machinery needed to run it unattended.

Key terms: snippets, pipeline, unattended, batch of RuckZone products, agents, single-purpose units, clear input, clear output, production, openai

12.1 From snippets to a pipeline

This section connects the six-stage product-enrichment workflow to the pipeline code needed to run, track, and review its results.

Key terms: Data Preparation, URL Discovery, HTML Fetching, Image Extraction, AI Extraction, Target Mapping, connective tissue, pipeline, agents, trusted output

12.1.1 What we mean by "agent"

Defines the chapter’s agents as small, typed pipeline functions and explains why ordinary code, rather than an autonomous LLM, controls their work.

Key terms: agent, single-responsibility function, typed contract, Pydantic model, orchestration, retries, validation, structured output

12.1.2 Why agents instead of one big function

This section explains why the scraping pipeline is split into agents rather than one large function, and why their shared data contract should be designed first.

Key terms: single responsibility, independent failure, reuse, recorded status, clear contracts, typed objects, hand-offs

12.2 Designing the data contract

This section defines typed data contracts for a product-enrichment pipeline, separating raw extraction from clean output and carrying each product’s state between agents.

Key terms: data contract, validated object, Pydantic model, ProductExtraction, EnrichedProduct, PipelineRecord, price_usd, weight_value, weight_unit, state

12.3 The four agents

This section introduces a four-agent pipeline that turns a search key into an enriched record and notes the shared infrastructure needed before implementing the agents.

Key terms: agent, search key, enriched record, data preparation, image extraction, shared infrastructure

12.3.1 Shared infrastructure: logging and retries

This section adds shared logging and selective retries so pipeline agents can report their work and withstand transient failures.

Key terms: configure_logging, with_retries, shared helpers, exponential backoff, network errors, rate-limited APIs

12.3.2 Agent 1: URL discovery

Agent 1 combines search and AI ranking behind a URL-discovery interface that handles retries internally and treats an empty search as a normal per-product outcome.

Key terms: search key, _search, with_retries, discover_url, LLM ranking, None, no candidates, clean failure, orchestrator

12.3.3 Agent 2: fetch and clean

Agent 2 fetches a product page, cleans it for extraction, and stops the pipeline when the result is unusable.

Key terms: Agent 2, extraction-ready text, ValueError, suspiciously empty, requests, extraction agent

12.3.4 Agent 3: AI extraction

Agent 3 wraps the Chapter 11 LLM extraction step as a pipeline agent with a deliberately limited retry policy.

Key terms: Agent 3, cleaned text, ProductExtraction, LLM, retry budget, two attempts, openai.OpenAIError, API errors

12.3.5 Agent 4: validate and map

This section shows how a deterministic validation agent converts extracted product data into a typed record and reports issues for human review.

Key terms: Stage 6, Target Mapping, ProductExtraction, EnrichedProduct, validate_and_map, issues, review queue, orchestrator, price_usd, range check

12.4 Orchestrating the pipeline

This section explains how an orchestrator runs the four scraping agents for one product, tracks where failures occur, and routes uncertain results to review.

Key terms: orchestrator, PipelineRecord, URL discovery, fetch and clean, AI extraction, validate and map, record.stage, stage, except block, needs_review

12.5 Running the pipeline end to end

The section shows how to run a product batch through the completed pipeline and interpret its results, review queue, and cost.

Key terms: batch, logging, run_pipeline, DataFrame, live web, validated table, review queue, ranking call, extraction call, SerpAPI search

12.6 Working the review queue

This section explains how a review queue keeps uncertain product records out of the catalog and how to tune the boundary between automatic acceptance and human review.

Key terms: review queue, save_results, status, success, needs_review, failed, trust boundary, validate_and_map, REQUIRED_FIELDS, weight_value

12.7 What you can reuse

The section presents the four-agent pipeline as a reusable pattern for turning messy sources into validated records while keeping uncertain decisions reviewable.

Key terms: discover_url, extract_product, validate_and_map, orchestrator, review queue, LLM, clear contracts, single-responsibility steps, explicit status, human in the loop

12.8 Lab

This lab extends the four-agent product-enrichment pipeline with fallback categorization and timing, then asks readers to report the results for five products.

12.9 Lab answers

The lab answers add guarded category inference and timing to the scraping pipeline, then use a pandas report to identify source-data gaps and performance bottlenecks.

Concepts explained in this chapter

← Chapter 11: AI-powered web scraping

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.