Chapter 12: AI-powered web scraping data pipelines in action
The chapter turns single-product web-scraping snippets into a batch pipeline built from single-purpose agents, with the supporting machinery needed to run it unattended.
After this chapter you will be able to
- From snippets to pipeline. Explain the gap between running separate scraping snippets on one product and running a pipeline unattended across hundreds of products with trustworthy output.
- Single-purpose agent design. Describe the planned agents as small units that each own one job, take a clear input, and return a clear output, then are chained into one pipeline with production-supporting machinery.
- Pipeline connective tissue. Explain that a pipeline is the code around the individual stage functions that runs them in order, handles failures, records what happened, and tracks records from raw input to trusted output or human review. Describe the section’s proposed shift from loose functions toward agents, without assuming a definition of agent that this section has not yet given.
- Typed single-purpose agents. Distinguish this chapter’s agents from autonomous LLM agents: each is a single-responsibility function with a declared input and output shape enforced by a Pydantic model. Some call an LLM internally; others are deterministic Python.
- Code-controlled pipeline responsibilities. Explain why ordinary code orchestrates the agents and controls flow, retries, and validation, while LLMs are reserved for reading messy unstructured content and producing structured output. Describe how this separation makes the pipeline debuggable, testable, and trustworthy.
- Modular pipeline benefits. Explain how separate agents make the pipeline easier to debug, test, recover, and reuse than one long function.
- Typed agent handoff contracts. Describe how typed inputs and outputs define the hand-offs between agents, and explain why the shared contract is designed before the agents.
- Typed pipeline handoffs. Explain why agents pass validated objects with named, typed fields rather than loosely shaped dictionaries, and describe the four models: pipeline input, LLM extraction, the final clean product, and a record that tracks progress.
- Extraction and storage models. Distinguish ProductExtraction, which reflects values as displayed on web pages, from EnrichedProduct, which holds typed values ready to store; describe the conversion as a separate pipeline stage.
- Product progress record. Describe how one PipelineRecord per product accumulates state across agents instead of requiring bare outputs to be stitched together, including the chosen URL, confidence, extraction, validation result, and failure location.
- Four-agent pipeline boundaries. Describe the pipeline as four agents, each responsible for one bounded stage of the work from search key to enriched record. Explain that data preparation builds search keys and image extraction is folded into extraction rather than treated as a separate agent.
- Shared pipeline helpers. Explain why logging and retries are defined once as helpers for every agent: pipeline steps should report what they are doing and survive transient failures rather than stop a batch.
- Selective exponential retry behavior. Describe how with_retries wraps a function to retry selected transient exceptions with a growing wait, while leaving programming bugs unretried.
- URL discovery agent interface. Describe how discover_url takes a search key, searches for candidate pages, and uses an LLM to select the best URL with a confidence indication. Explain that the network search is a private helper with retries, so callers use the public agent without managing retries.
- Empty search result handling. Explain that discover_url returns None when search finds no candidates rather than raising an error, because no results is a normal outcome that the orchestrator treats as a clean failure for that product.
- Fetch and clean handoff. Describe how Agent 2 takes a URL and returns clean, extraction-ready text for the extraction agent, or raises an exception if the page cannot be retrieved.
- Suspiciously empty page guard. Explain why Agent 2 raises a ValueError when cleaned text is suspiciously empty: the page may be blocked or rely on JavaScript that requests cannot execute, and passing empty text to the extraction agent would waste an LLM call and invite hallucination.
- AI extraction agent contract. Describe how Agent 3 takes cleaned text and returns a ProductExtraction, wrapping the earlier LLM extraction step as an agent.
- Limited extraction retry policy. Explain that Agent 3 allows two attempts rather than three and retries only openai.OpenAIError exceptions, because LLM extraction is the pipeline’s most expensive call and bad inputs should not trigger repeated paid attempts.
- Deterministic target mapping. Explain that Stage 6 uses Python, not an LLM, to turn a messy ProductExtraction into a clean, typed EnrichedProduct by normalizing values and checking for problems.
- Issue reporting and review boundary. Describe the agent's two return values—a clean record and a list of issues—and explain how that list determines whether a record needs human review while the orchestrator decides what to do next.
- Ordered agent orchestration. Describe how the orchestrator runs URL discovery, fetch and clean, AI extraction, and validate and map in order for one product, passing state through a PipelineRecord and interpreting the outcome.
- Stage-based failure diagnosis. Explain why the orchestrator sets record.stage before each agent runs and how the exception handler uses it to identify where a failure happened.
- Three-way outcome routing. Distinguish success, outright failure, and needs_review, and explain why low-confidence or issue-flagged AI results go to a review queue rather than straight into the database.
- End-to-end batch run. Describe how to run a batch by defining the products, enabling logging, and calling run_pipeline, then explain what the log and final DataFrame report.
- Honest handling of uncertain results. Explain why live-web results vary between runs and why a useful pipeline reports validated results alongside an honest count of items it could not confidently resolve.
- Review queue and save filter. Explain why records marked needs_review or failed are held back while save_results writes only records with status == "success" to the CSV destined for the catalog.
- Trust boundary tuning. Describe how stricter price validation, requiring weight_value, or routing medium-confidence results to review changes which products need human attention. Justify the chosen boundary in terms of the cost of admitting a wrong record.
- Reusable four-agent pipeline. Explain how the discover, fetch, extract, and validate agents can be reused beyond product pages by changing the input source and extraction schema while retaining most validation and orchestration logic.
- Bounded AI and human review. Justify using the LLM for extraction while predictable code handles normalization, validation, and status tracking, with uncertain results routed to human review. Explain how clear contracts and single-responsibility steps support safer systems even if a model later makes more decisions.
What it builds on
- Scaling Extraction Across Multiple Sites (chapter 11) · Scaling extraction beyond a single site or result.
- AI-Assisted URL Discovery (chapter 11) · URL discovery as an existing piece of the workflow.
- Smarter HTML Cleaning (chapter 11) · HTML cleaning as an existing piece of the workflow.
- From Manual Selectors to AI Extraction (chapter 11) · AI extraction as an existing piece of the workflow.
- Finding Candidate URLs Programmatically (chapter 11) · Programmatic candidate-URL discovery uses the search API.
- The Enrichment Pipeline (chapter 10) · Introduces the enrichment pipeline.
- Mapping AI to the Enrichment Pipeline (chapter 11) · Maps AI to stages of the enrichment pipeline.
- Defining What You Want (chapter 11) · The earlier ProductExtraction Pydantic model introduced a defined shape for extracted product data.
Section by section
Introduction
The chapter turns single-product web-scraping snippets into a batch pipeline built from single-purpose agents, with the supporting machinery needed to run it unattended.
- From snippets to pipeline: Explain the gap between running separate scraping snippets on one product and running a pipeline unattended across hundreds of products with trustworthy output.
Why it matters: Working code is only a starting point if the results still require someone to run each step and inspect them by eye. - Single-purpose agent design: Describe the planned agents as small units that each own one job, take a clear input, and return a clear output, then are chained into one pipeline with production-supporting machinery.
Why it matters: Clear boundaries between steps make it possible to assemble the existing functions into a pipeline that can keep running. - Pipeline environment prerequisites: State that the chapter requires openai, pydantic, requests, beautifulsoup4, and pandas, plus OpenAI API and SerpAPI keys made available through the environment before running the listings.
Why it matters: The pipeline’s API calls need their keys available before the code runs.
Key terms: snippets, pipeline, unattended, batch of RuckZone products, agents, single-purpose units, clear input, clear output, production, openai
12.1 From snippets to a pipeline
This section connects the six-stage product-enrichment workflow to the pipeline code needed to run, track, and review its results.
- Six-stage enrichment workflow: Describe the six stages in order: prepare product data, discover a URL, fetch and clean HTML, extract images, use AI to extract structured fields, and map and validate the final record. State that Chapter 11 supplied working code for URL discovery, HTML fetching, and AI extraction, but not the complete pipeline.
Why it matters: The stages show what the pipeline must connect, including work not covered by the earlier snippets. - Pipeline connective tissue: Explain that a pipeline is the code around the individual stage functions that runs them in order, handles failures, records what happened, and tracks records from raw input to trusted output or human review. Describe the section’s proposed shift from loose functions toward agents, without assuming a definition of agent that this section has not yet given.
Why it matters: Working extraction snippets are not enough unless their outcomes and failures can be managed across a whole record.
Key terms: Data Preparation, URL Discovery, HTML Fetching, Image Extraction, AI Extraction, Target Mapping, connective tissue, pipeline, agents, trusted output
12.1.1 What we mean by "agent"
Defines the chapter’s agents as small, typed pipeline functions and explains why ordinary code, rather than an autonomous LLM, controls their work.
- Typed single-purpose agents: Distinguish this chapter’s agents from autonomous LLM agents: each is a single-responsibility function with a declared input and output shape enforced by a Pydantic model. Some call an LLM internally; others are deterministic Python.
Why it matters: This definition makes it clear what each pipeline step can accept and produce. - Code-controlled pipeline responsibilities: Explain why ordinary code orchestrates the agents and controls flow, retries, and validation, while LLMs are reserved for reading messy unstructured content and producing structured output. Describe how this separation makes the pipeline debuggable, testable, and trustworthy.
Why it matters: Separating language-model work from predictable pipeline control makes failures easier to find and manage.
Key terms: agent, single-responsibility function, typed contract, Pydantic model, orchestration, retries, validation, structured output
12.1.2 Why agents instead of one big function
This section explains why the scraping pipeline is split into agents rather than one large function, and why their shared data contract should be designed first.
- Modular pipeline benefits: Explain how separate agents make the pipeline easier to debug, test, recover, and reuse than one long function.
Why it matters: Separating responsibilities keeps a problem with one product or step from derailing the whole pipeline. - Typed agent handoff contracts: Describe how typed inputs and outputs define the hand-offs between agents, and explain why the shared contract is designed before the agents.
Why it matters: A clear contract lets you reason about how steps connect without running the entire pipeline.
Key terms: single responsibility, independent failure, reuse, recorded status, clear contracts, typed objects, hand-offs
12.2 Designing the data contract
This section defines typed data contracts for a product-enrichment pipeline, separating raw extraction from clean output and carrying each product’s state between agents.
- Typed pipeline handoffs: Explain why agents pass validated objects with named, typed fields rather than loosely shaped dictionaries, and describe the four models: pipeline input, LLM extraction, the final clean product, and a record that tracks progress.
Why it matters: A clear contract lets each agent focus on its job instead of repeatedly checking the shape of its input. - Extraction and storage models: Distinguish ProductExtraction, which reflects values as displayed on web pages, from EnrichedProduct, which holds typed values ready to store; describe the conversion as a separate pipeline stage.
Why it matters: Keeping raw page values separate from clean stored values makes the transformation explicit. - Product progress record: Describe how one PipelineRecord per product accumulates state across agents instead of requiring bare outputs to be stitched together, including the chosen URL, confidence, extraction, validation result, and failure location.
Why it matters: The accumulated record shows both the product result and what happened along the way.
Key terms: data contract, validated object, Pydantic model, ProductExtraction, EnrichedProduct, PipelineRecord, price_usd, weight_value, weight_unit, state
12.3 The four agents
This section introduces a four-agent pipeline that turns a search key into an enriched record and notes the shared infrastructure needed before implementing the agents.
- Four-agent pipeline boundaries: Describe the pipeline as four agents, each responsible for one bounded stage of the work from search key to enriched record. Explain that data preparation builds search keys and image extraction is folded into extraction rather than treated as a separate agent.
Why it matters: These boundaries make it easier to follow how the agents fit together. - Shared agent infrastructure: Explain that a small piece of shared infrastructure must be put in place before writing the agents because every agent will rely on it.
Why it matters: It explains why the implementation begins with common support rather than an individual agent.
Key terms: agent, search key, enriched record, data preparation, image extraction, shared infrastructure
12.3.1 Shared infrastructure: logging and retries
This section adds shared logging and selective retries so pipeline agents can report their work and withstand transient failures.
- Shared pipeline helpers: Explain why logging and retries are defined once as helpers for every agent: pipeline steps should report what they are doing and survive transient failures rather than stop a batch.
Why it matters: Shared helpers give each agent consistent visibility and resilience without repeating the same code. - Selective exponential retry behavior: Describe how with_retries wraps a function to retry selected transient exceptions with a growing wait, while leaving programming bugs unretried.
Why it matters: Backing off gives a flaky or rate-limited service time to recover without repeatedly piling on requests.
Key terms: configure_logging, with_retries, shared helpers, exponential backoff, network errors, rate-limited APIs
12.3.2 Agent 1: URL discovery
Agent 1 combines search and AI ranking behind a URL-discovery interface that handles retries internally and treats an empty search as a normal per-product outcome.
- URL discovery agent interface: Describe how discover_url takes a search key, searches for candidate pages, and uses an LLM to select the best URL with a confidence indication. Explain that the network search is a private helper with retries, so callers use the public agent without managing retries.
Why it matters: This interface lets the pipeline use URL discovery without depending on its search and retry details. - Empty search result handling: Explain that discover_url returns None when search finds no candidates rather than raising an error, because no results is a normal outcome that the orchestrator treats as a clean failure for that product.
Why it matters: Distinguishing an absent result from an exception keeps one unsuccessful product search from being treated as a pipeline error.
Key terms: search key, _search, with_retries, discover_url, LLM ranking, None, no candidates, clean failure, orchestrator
12.3.3 Agent 2: fetch and clean
Agent 2 fetches a product page, cleans it for extraction, and stops the pipeline when the result is unusable.
- Fetch and clean handoff: Describe how Agent 2 takes a URL and returns clean, extraction-ready text for the extraction agent, or raises an exception if the page cannot be retrieved.
Why it matters: This handoff keeps page retrieval and cleaning separate from AI extraction. - Suspiciously empty page guard: Explain why Agent 2 raises a ValueError when cleaned text is suspiciously empty: the page may be blocked or rely on JavaScript that requests cannot execute, and passing empty text to the extraction agent would waste an LLM call and invite hallucination.
Why it matters: Failing at the fetch stage makes an unusable page visible before it costs an extraction call.
Key terms: Agent 2, extraction-ready text, ValueError, suspiciously empty, requests, extraction agent
12.3.4 Agent 3: AI extraction
Agent 3 wraps the Chapter 11 LLM extraction step as a pipeline agent with a deliberately limited retry policy.
- AI extraction agent contract: Describe how Agent 3 takes cleaned text and returns a ProductExtraction, wrapping the earlier LLM extraction step as an agent.
Why it matters: This identifies what the extraction agent receives and passes to the next pipeline stage. - Limited extraction retry policy: Explain that Agent 3 allows two attempts rather than three and retries only openai.OpenAIError exceptions, because LLM extraction is the pipeline’s most expensive call and bad inputs should not trigger repeated paid attempts.
Why it matters: A narrow retry policy limits unnecessary costs without giving up retries for genuine API errors.
Key terms: Agent 3, cleaned text, ProductExtraction, LLM, retry budget, two attempts, openai.OpenAIError, API errors
12.3.5 Agent 4: validate and map
This section shows how a deterministic validation agent converts extracted product data into a typed record and reports issues for human review.
- Deterministic target mapping: Explain that Stage 6 uses Python, not an LLM, to turn a messy ProductExtraction into a clean, typed EnrichedProduct by normalizing values and checking for problems.
Why it matters: Keeping straightforward conversions in code makes the pipeline faster and easier to test. - Issue reporting and review boundary: Describe the agent's two return values—a clean record and a list of issues—and explain how that list determines whether a record needs human review while the orchestrator decides what to do next.
Why it matters: Separating problem detection from decisions about handling problems makes the review queue possible. - Standalone validation test cases: Describe how to test validate_and_map with hand-built ProductExtraction objects before connecting the agents: check a complete product, a missing price, and a zero price.
Why it matters: These quick tests verify normal conversion and error reporting without a network or LLM dependency.
Key terms: Stage 6, Target Mapping, ProductExtraction, EnrichedProduct, validate_and_map, issues, review queue, orchestrator, price_usd, range check
12.4 Orchestrating the pipeline
This section explains how an orchestrator runs the four scraping agents for one product, tracks where failures occur, and routes uncertain results to review.
- Ordered agent orchestration: Describe how the orchestrator runs URL discovery, fetch and clean, AI extraction, and validate and map in order for one product, passing state through a PipelineRecord and interpreting the outcome.
Why it matters: The orchestrator turns separate agents into a pipeline that can process a product end to end. - Stage-based failure diagnosis: Explain why the orchestrator sets record.stage before each agent runs and how the exception handler uses it to identify where a failure happened.
Why it matters: Knowing the failed stage helps distinguish a fetch problem from an extraction problem across a batch. - Three-way outcome routing: Distinguish success, outright failure, and needs_review, and explain why low-confidence or issue-flagged AI results go to a review queue rather than straight into the database.
Why it matters: Reviewing uncertain results prevents plausible but wrong AI output from silently contaminating the catalog.
Key terms: orchestrator, PipelineRecord, URL discovery, fetch and clean, AI extraction, validate and map, record.stage, stage, except block, needs_review
12.5 Running the pipeline end to end
The section shows how to run a product batch through the completed pipeline and interpret its results, review queue, and cost.
- End-to-end batch run: Describe how to run a batch by defining the products, enabling logging, and calling run_pipeline, then explain what the log and final DataFrame report.
Why it matters: The run turns the assembled pipeline into results you can inspect product by product. - Honest handling of uncertain results: Explain why live-web results vary between runs and why a useful pipeline reports validated results alongside an honest count of items it could not confidently resolve.
Why it matters: A visible review queue keeps uncertain product data from being mistaken for trustworthy data. - Batch run cost estimate: State the section’s approximate costs for a five-product test and the full 450-product spreadsheet, and identify the calls included in the estimate.
Why it matters: A small trial run provides a practical cost check before processing the full spreadsheet.
Key terms: batch, logging, run_pipeline, DataFrame, live web, validated table, review queue, ranking call, extraction call, SerpAPI search
12.6 Working the review queue
This section explains how a review queue keeps uncertain product records out of the catalog and how to tune the boundary between automatic acceptance and human review.
- Review queue and save filter: Explain why records marked needs_review or failed are held back while save_results writes only records with status == "success" to the CSV destined for the catalog.
Why it matters: The status filter lets the pipeline run unattended without silently admitting untrusted records to the catalog. - Trust boundary tuning: Describe how stricter price validation, requiring weight_value, or routing medium-confidence results to review changes which products need human attention. Justify the chosen boundary in terms of the cost of admitting a wrong record.
Why it matters: There is no universally correct queue size: stricter rules catch more questionable records but require more human review.
Key terms: review queue, save_results, status, success, needs_review, failed, trust boundary, validate_and_map, REQUIRED_FIELDS, weight_value
12.7 What you can reuse
The section presents the four-agent pipeline as a reusable pattern for turning messy sources into validated records while keeping uncertain decisions reviewable.
- Reusable four-agent pipeline: Explain how the discover, fetch, extract, and validate agents can be reused beyond product pages by changing the input source and extraction schema while retaining most validation and orchestration logic.
Why it matters: The design can be adapted to new sources and record types without rebuilding the whole pipeline. - Bounded AI and human review: Justify using the LLM for extraction while predictable code handles normalization, validation, and status tracking, with uncertain results routed to human review. Explain how clear contracts and single-responsibility steps support safer systems even if a model later makes more decisions.
Why it matters: Clear boundaries make unstructured-data pipelines more trustworthy and easier to review.
Key terms: discover_url, extract_product, validate_and_map, orchestrator, review queue, LLM, clear contracts, single-responsibility steps, explicit status, human in the loop
12.8 Lab
This lab extends the four-agent product-enrichment pipeline with fallback categorization and timing, then asks readers to report the results for five products.
12.9 Lab answers
The lab answers add guarded category inference and timing to the scraping pipeline, then use a pandas report to identify source-data gaps and performance bottlenecks.
Concepts explained in this chapter
- PipelineRecordPipelineRecord is a structured model that wraps a product with bookkeeping fields for tracking pipeline progress, confidence, extracted data, validation, and failure state.
- review queueA review queue is an ordered collection of items awaiting inspection or resolution by a reviewer.
- status trackingStatus tracking is the recording and monitoring of the current state of tasks, records, or processing outcomes.
- data contractA data contract is a formal specification of the structure, fields, types, and validity requirements that data exchanged between processing components must satisfy.
- product extractionProduct extraction is the process of extracting product attributes from source content.
- PydanticA Pydantic model is a structured data class defined with Pydantic fields and type annotations that can validate and serialize values.
- data frameA pandas DataFrame is a two-dimensional labeled tabular data structure in the pandas library.
- structured extractionAI-powered extraction is the use of an artificial intelligence model to identify and retrieve requested fields from source content based on field descriptions rather than fixed location rules.
- url discoveryURL discovery is the process of finding and selecting a web address for a particular product or other target item.
- data cleaningData cleaning is the process of detecting and correcting, removing, or standardizing inaccurate, incomplete, duplicated, or unsuitable data.
- retryRetry logic is a control technique that repeats a failed operation according to defined attempt and delay policies.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
