Chapter 11: AI-powered web scraping
This introduction explains why the manual RuckZone scraping approach does not scale and previews an incremental, stage-by-stage use of AI in the enrichment pipeline.
After this chapter you will be able to
- Incremental AI enrichment roadmap. Describe the chapter’s plan to inspect each stage of the existing RuckZone enrichment pipeline for places AI can replace, augment, or extend manual work, rather than rebuild it from scratch. State that the chapter will cover AI-assisted URL discovery, HTML cleaning, product extraction, comparison with manual extraction, token costs, and a reusable opportunity checklist.
- Enrichment pipeline stages. Describe the six stages in order: data preparation, URL discovery, HTML fetching, image extraction, AI extraction, and target mapping. Explain that the earlier manual work loaded product data and search keys, fetched and parsed HTML, and used site-specific selectors for titles and prices.
- Manual scraping pain points. Explain why manual scraping becomes difficult beyond a few pages: selectors must be maintained per site, variants on one URL require choosing the right product, HTML layouts vary, and finding a trustworthy product URL takes manual judgment. Relate these problems to why AI could help.
- AI opportunity test. Explain how to decide whether a pipeline task is a good candidate for an LLM: a person could perform it repeatedly from clear rules without specialized tools, especially when reading unstructured content into a defined schema. Distinguish such tasks from steps better handled by deterministic code, simple rules, or SQL.
- Extracted data and AI extraction. Explain that extracted data already exists in a source but may be buried in HTML or other page content, and describe how an LLM can pull specified fields from raw or cleaned content using a schema and prompt instead of site-specific selectors.
- Enriched versus extracted fields. Distinguish enriched data from extracted data: enriched fields are not explicitly present in the source but are inferred or generated from it, or added from an external source.
- Judgment and validation of enrichment. Explain that enriched fields require judgment rather than just reading comprehension, making them useful but less reliable and in need of more careful validation than extracted fields.
- Safe uses and production risks. Explain why synthetic data suits testing, development, schema validation, and load testing but not production enrichment or customer-facing records where accuracy matters. Describe the risk of treating generated content as ground truth and why the chapter instead focuses mainly on extracted data.
- Selective AI use in pipelines. Explain why AI should be reserved for pipeline work that is unstructured, variable, or requires judgment, rather than added to every stage.
- Highest payoff pipeline stages. State that stages 2, 3, and 5 have the highest AI payoff because manual approaches break down at scale, and that these stages are the chapter's focus.
- Two-step URL discovery approach. Describe the proposed automation: use a search API to find candidate product URLs, then use an LLM to rank those candidates.
- Programmatic URL candidate discovery. Explain how a search API such as SerpAPI takes a product query and returns candidate pages with titles, URLs, snippets, and search positions. Describe these results as candidates for LLM ranking, not as an already chosen source.
- Product source ranking criteria. Explain why search-result relevance differs from suitability for product-data extraction, and describe the criteria the LLM uses to choose a source.
- Structured URL ranking response. Describe the URLRanking response contract and how the system prompt guides the model's choice.
- URL selection review workflow. Trace the two-step workflow from programmatic candidate search through AI ranking, and explain how to check the chosen URL before trusting it.
- Pre-extraction cleaning benefits. Explain why removing non-product content before an LLM extraction call reduces token costs, response time, and distracting input.
- Two-layer HTML filtering. Describe the improved BeautifulSoup approach: remove tags that are usually irrelevant to product extraction, then filter generic containers using class-name patterns associated with non-product regions. Distinguish these reusable rules from AI-based cleaning.
- Manual Selectors Versus AI Extraction. Explain why manually written CSS selectors require inspecting each site's HTML and can break when it changes. Describe how AI extraction instead specifies the desired fields in a schema and prompt, leaving the model to locate them.
- Product extraction schema contract. Explain how the ProductExtraction Pydantic model acts as a contract between the pipeline and the LLM by specifying the fields, their types, and their meanings. Name the product data it targets and distinguish the required fields from the optional ones.
- Field descriptions as model guidance. Explain that Field(description=...) annotations both document or validate the expected output for code and guide the model: with OpenAI structured output, their descriptions appear in the schema the model sees and can specify what to find and how to format it.
- Schema-based AI extraction. Explain that a function call sends cleaned HTML and a Pydantic schema to an LLM, which finds product information in the page text and returns structured data without site-specific selectors, regex, or HTML-specific branches.
- Prompt rules for reliable fields. Explain why the prompt requires fields to appear explicitly on the page, uses null rather than guesses, and distinguishes sale prices from original prices.
- Cross-site extraction tradeoffs. Justify using AI extraction across many sites despite possible hallucinations, per-call cost, and slower execution than regex: it avoids writing custom extraction code for each site.
- Extraction results on one page. Describe what the manual and AI approaches extracted from the same product page: both found the price, but AI found additional fields and a more complete product name.
- Cross-site extraction scalability. Explain why the manual extractor would require site-specific code for a catalog spanning dozens of sites, whereas the AI approach can reuse one extraction function.
- Batch extraction workflow. Trace the four steps repeated for each product: find candidate URLs, have the model choose a page, fetch and clean its HTML, and extract data from the cleaned content. Explain that the workflow records each product’s result and does not stop the whole batch when one product fails.
- Extraction failures and result statuses. Explain why an extraction may fail or return incomplete data, and distinguish success, error, and a possible partial status in the results.
- Production failure handling measures. Describe the additional measures proposed for a production extraction pipeline: retries for transient failures, logging every extraction attempt, and a review queue for low-confidence results.
- Token-based extraction costs. Explain that LLM APIs charge separately for input and output tokens, with output tokens typically costing more per token. Describe using a token-count and cost estimate before a large batch run rather than judging affordability from a few notebook examples.
- Cleaning and model selection. Explain how aggressive HTML cleaning reduces input tokens and why simpler tasks such as URL ranking and content triage can use a cheaper model while extraction may warrant a more expensive one.
- AI task suitability. Explain how input structure, source-specific rules, and the “smart intern test” help identify steps suited to AI: tasks involving messy inputs and human-like judgment are better candidates than fixed operations or tasks requiring precise calculation, specialist expertise, or external access.
- Cost and deterministic alternatives. Describe how to estimate AI’s per-record cost against the value it adds and alternatives such as regex, dictionaries, SQL joins, or ordinary code. Explain why simple transformations, exact lookups, and unjustifiably expensive high-volume decisions should not be sent to an LLM.
- Validation and failure planning. Explain why an AI step needs ways to check its output and a plan for inevitable errors. Describe checks such as schema validation, spot-sampling, known-value comparisons, or confidence scoring, alongside status tracking, review queues, and fallback logic; state that safety-critical outputs must not be used without human review as the sole basis for decisions.
What it builds on
- Manual Extraction Across Multiple Sites (chapter 10) · Manual extraction across multiple sites and its site-specific logic.
- The Enrichment Pipeline (chapter 10) · The existing product enrichment pipeline that this chapter revisits.
- Loading the Product Data (chapter 10) · Loading product data.
- Web Scraping Fundamentals (chapter 10) · Fetching and parsing HTML for manual scraping.
- Finding a Product URL Manually (chapter 10) · Manually finding and evaluating product pages.
- Parsing and cleaning HTML with BeautifulSoup (chapter 10) · Parsing HTML to find page content.
- Search for the product page (chapter 10) · Manual searching for a product page
- Evaluate the results and choose a source (chapter 10) · Evaluating search results and choosing a source
Section by section
Introduction
This introduction explains why the manual RuckZone scraping approach does not scale and previews an incremental, stage-by-stage use of AI in the enrichment pipeline.
- Manual scraping bottlenecks: Explain why the earlier hand-built scraping approach works but becomes difficult to scale: different site structures and product variants require custom selectors, fallback logic, and edge-case handling.
Why it matters: These bottlenecks show where AI assistance might be useful without assuming every manual step should be replaced. - Incremental AI enrichment roadmap: Describe the chapter’s plan to inspect each stage of the existing RuckZone enrichment pipeline for places AI can replace, augment, or extend manual work, rather than rebuild it from scratch. State that the chapter will cover AI-assisted URL discovery, HTML cleaning, product extraction, comparison with manual extraction, token costs, and a reusable opportunity checklist.
Why it matters: The stage-by-stage approach connects the existing scraper to the more production-ready pipeline built next.
Key terms: BeautifulSoup, CSS selectors, variant-specific logic, fallback strategies, RuckZone enrichment pipeline, AI opportunities, AI-assisted URL discovery, HTML cleaning, product extraction, token costs
11.1 Where we left off
Recaps the six-stage product enrichment pipeline and the manual scraping problems that motivate AI assistance.
- Enrichment pipeline stages: Describe the six stages in order: data preparation, URL discovery, HTML fetching, image extraction, AI extraction, and target mapping. Explain that the earlier manual work loaded product data and search keys, fetched and parsed HTML, and used site-specific selectors for titles and prices.
Why it matters: The stages show where the existing work fits and where AI assistance can enter the pipeline. - Manual scraping pain points: Explain why manual scraping becomes difficult beyond a few pages: selectors must be maintained per site, variants on one URL require choosing the right product, HTML layouts vary, and finding a trustworthy product URL takes manual judgment. Relate these problems to why AI could help.
Why it matters: These recurring difficulties motivate the AI-assisted methods introduced in this chapter.
Key terms: enrichment pipeline, Data Preparation, URL Discovery, HTML Fetching, Image Extraction, AI Extraction, selector treadmill, variant pages, inconsistent HTML, manual URL selection
11.2 Recognizing AI Opportunities in Data Pipelines
Introduces a test for spotting pipeline tasks suited to AI and names three categories of AI-generated data.
- AI opportunity test: Explain how to decide whether a pipeline task is a good candidate for an LLM: a person could perform it repeatedly from clear rules without specialized tools, especially when reading unstructured content into a defined schema. Distinguish such tasks from steps better handled by deterministic code, simple rules, or SQL.
Why it matters: This test helps you choose where AI adds value rather than using it at every pipeline step. - AI-generated data categories: State the three categories of AI-generated data named here: extracted data, enriched data, and synthetic data.
Why it matters: These names organize the kinds of AI output the following subsections examine.
Key terms: deterministic code, unstructured content, structured fields, schema, extracted data, enriched data, synthetic data
11.2.1 Extracted Data
This section defines extracted data and introduces LLM extraction as a replacement for site-specific CSS selectors.
- Extracted data and AI extraction: Explain that extracted data already exists in a source but may be buried in HTML or other page content, and describe how an LLM can pull specified fields from raw or cleaned content using a schema and prompt instead of site-specific selectors.
Why it matters: This distinction explains the technique used to extract product information across different sites.
Key terms: extracted data, AI extraction, schema, prompt, CSS-selector-based scraping
11.2.2 Enriched Data
This section distinguishes enriched data from extracted data and explains why inferred fields need more careful validation.
- Enriched versus extracted fields: Distinguish enriched data from extracted data: enriched fields are not explicitly present in the source but are inferred or generated from it, or added from an external source.
Why it matters: This distinction helps you decide whether a pipeline step is reading a field or making a judgment about it. - Judgment and validation of enrichment: Explain that enriched fields require judgment rather than just reading comprehension, making them useful but less reliable and in need of more careful validation than extracted fields.
Why it matters: Treating an inferred field as a directly observed fact can let an unsupported claim enter a catalog.
Key terms: enriched data, extraction, external sources, judgment, enriched fields, validate
11.2.3 Synthetic Data
The section defines synthetic data, explains where it is useful and risky, and distinguishes it from the chapter’s focus on extracting and enriching real product data.
- Synthetic data versus source data: Distinguish synthetic data from extracted or enriched data: it is generated from scratch rather than taken or inferred from a specific source. Give an example such as schema-matching test records or mock datasets.
Why it matters: This distinction helps you tell generated examples apart from information about real products. - Safe uses and production risks: Explain why synthetic data suits testing, development, schema validation, and load testing but not production enrichment or customer-facing records where accuracy matters. Describe the risk of treating generated content as ground truth and why the chapter instead focuses mainly on extracted data.
Why it matters: Keeping generated examples out of factual records prevents plausible fiction from becoming production data.
Key terms: synthetic data, extracted data, enriched data, production enrichment, ground truth, schema validation, load testing
11.3 Mapping AI to the Enrichment Pipeline
This section maps AI onto the six-stage web-scraping enrichment pipeline and identifies where it is worth using.
- Selective AI use in pipelines: Explain why AI should be reserved for pipeline work that is unstructured, variable, or requires judgment, rather than added to every stage.
Why it matters: Choosing AI selectively avoids unnecessary cost, latency, and complexity. - Highest payoff pipeline stages: State that stages 2, 3, and 5 have the highest AI payoff because manual approaches break down at scale, and that these stages are the chapter's focus.
Why it matters: These priorities explain where the chapter concentrates its AI-assisted scraping work.
Key terms: enrichment pipeline, Stage 1, stages 2, 3, and 5, manual approaches, AI payoff
11.4 AI-Assisted URL Discovery
This section introduces a two-step way to replace slow, inconsistent manual product URL selection with search API candidates ranked by an LLM.
- Manual URL discovery drawbacks: Explain why manually searching for product URLs is slow and can produce inconsistent choices when different people see or evaluate results differently.
Why it matters: These drawbacks explain why the chapter automates URL discovery. - Two-step URL discovery approach: Describe the proposed automation: use a search API to find candidate product URLs, then use an LLM to rank those candidates.
Why it matters: It separates finding possible pages from deciding which are most promising.
Key terms: product URLs, manually, inconsistent, search API, candidate URLs, LLM, rank
11.4.1 Finding Candidate URLs Programmatically
This section replaces manual product-page searching with a search API that returns candidate URLs for later AI ranking.
- Programmatic URL candidate discovery: Explain how a search API such as SerpAPI takes a product query and returns candidate pages with titles, URLs, snippets, and search positions. Describe these results as candidates for LLM ranking, not as an already chosen source.
Why it matters: Finding candidates programmatically gives the ranking step a consistent list of pages to evaluate. - Search access and manual fallback: Describe how to supply a SerpAPI key through the SERPAPI_KEY environment variable or a .env file, and how to continue without a key by manually assembling three to five candidate URLs for the ranking function.
Why it matters: The manual fallback lets a reader try the ranking step even without search API access.
Key terms: SerpAPI, search API, candidate URLs, snippet, search position, SERPAPI_KEY, .env file, candidates list, ranking function
11.4.2 Ranking URLs with AI
This section uses an LLM to rank search-result URLs for product-data extraction, then checks the structured choice before using it downstream.
- Product source ranking criteria: Explain why search-result relevance differs from suitability for product-data extraction, and describe the criteria the LLM uses to choose a source.
Why it matters: A highly ranked search result may be a poor source of the product fields your pipeline needs. - Structured URL ranking response: Describe the URLRanking response contract and how the system prompt guides the model's choice.
Why it matters: A defined decision rule and response shape make the model's selection easier to validate, review, and pass to the next pipeline step. - URL selection review workflow: Trace the two-step workflow from programmatic candidate search through AI ranking, and explain how to check the chosen URL before trusting it.
Why it matters: Review catches plausible-looking choices that may not actually serve the extraction pipeline.
Key terms: candidate URLs, product-detail pages, manufacturer or brand pages, structured product data, URLRanking, Pydantic data class, system prompt, best_url, confidence, reasoning
11.5 Smarter HTML Cleaning
This section shows how to shrink product-page HTML with rules-based cleaning before sending it to an LLM for extraction.
- Pre-extraction cleaning benefits: Explain why removing non-product content before an LLM extraction call reduces token costs, response time, and distracting input.
Why it matters: Cleaning the page before the AI call can make extraction cheaper and more focused. - Two-layer HTML filtering: Describe the improved BeautifulSoup approach: remove tags that are usually irrelevant to product extraction, then filter generic containers using class-name patterns associated with non-product regions. Distinguish these reusable rules from AI-based cleaning.
Why it matters: Tags alone miss distractions that sites place inside ordinary containers. - Cleaning comparison and residual review: Describe how to compare basic and aggressive cleaning on the same product page using character counts, then inspect the remaining paragraphs to decide which contain product data and whether further simple rules suffice or AI triage is needed.
Why it matters: A smaller page is useful only if the remaining text still supports product extraction.
Key terms: cleaned HTML, tokens, LLM extraction, BeautifulSoup, tag-level HTML cleaning, class-name patterns, rules-based, basic cleaning, aggressive cleaning, character counts
11.6 From Manual Selectors to AI Extraction
This section contrasts brittle, site-specific CSS selectors with AI extraction that uses a schema and prompt to specify desired fields.
- Manual Selectors Versus AI Extraction: Explain why manually written CSS selectors require inspecting each site's HTML and can break when it changes. Describe how AI extraction instead specifies the desired fields in a schema and prompt, leaving the model to locate them.
Why it matters: This shift changes the extraction task from finding fields in a page to defining the output you want.
Key terms: CSS selectors, AI extraction, schema, prompt
11.6.1 Defining What You Want
This section defines a Pydantic product-extraction schema that guides the model’s output while accommodating incomplete product pages.
- Product extraction schema contract: Explain how the ProductExtraction Pydantic model acts as a contract between the pipeline and the LLM by specifying the fields, their types, and their meanings. Name the product data it targets and distinguish the required fields from the optional ones.
Why it matters: A clear contract gives the pipeline predictable output without requiring every product page to contain every detail. - Field descriptions as model guidance: Explain that Field(description=...) annotations both document or validate the expected output for code and guide the model: with OpenAI structured output, their descriptions appear in the schema the model sees and can specify what to find and how to format it.
Why it matters: Useful field descriptions can replace extra prompt instructions while making the expected output clearer. - Preserving displayed price and weight: Justify defining price and weight as strings during extraction so displayed values retain their currency symbols or units, then parsing and normalizing them later with deterministic code in Stage 6, Target Mapping.
Why it matters: Preserving the source format avoids losing information before a controlled normalization step.
Key terms: ProductExtraction, Pydantic model, target schema, product_name, brand_name, Field(description=...), response_format, structured output, implicit prompt engineering, price
11.6.2 Letting AI Do the Extraction
The section shows how an LLM can extract schema-shaped product data from cleaned HTML, and explains the prompt rules and tradeoffs that make the approach useful across sites.
- Schema-based AI extraction: Explain that a function call sends cleaned HTML and a Pydantic schema to an LLM, which finds product information in the page text and returns structured data without site-specific selectors, regex, or HTML-specific branches.
Why it matters: This is how the same extraction approach can work across pages with different HTML structures. - Prompt rules for reliable fields: Explain why the prompt requires fields to appear explicitly on the page, uses null rather than guesses, and distinguishes sale prices from original prices.
Why it matters: These rules reduce invented values and pricing ambiguity before the data reaches downstream code. - Cross-site extraction tradeoffs: Justify using AI extraction across many sites despite possible hallucinations, per-call cost, and slower execution than regex: it avoids writing custom extraction code for each site.
Why it matters: The approach is practical when reducing cross-site maintenance outweighs its cost, latency, and reliability risks.
Key terms: cleaned HTML, LLM, Pydantic schema, structured product data, explicit presence, null, sale price, original price, hallucinating fields, hallucinate
11.6.3 Manual vs. AI: A Side-by-Side Comparison
A side-by-side product-page comparison shows that AI extraction captures more information than a manual selector and can reuse one function across many sites.
- Extraction results on one page: Describe what the manual and AI approaches extracted from the same product page: both found the price, but AI found additional fields and a more complete product name.
Why it matters: This comparison shows where AI adds value and where a simple selector already works. - Cross-site extraction scalability: Explain why the manual extractor would require site-specific code for a catalog spanning dozens of sites, whereas the AI approach can reuse one extraction function.
Why it matters: The need to process 450 products makes reusable extraction more valuable than a single-page improvement.
Key terms: manual approach, AI approach, selectors, GR1, GR1 26L, extract_manual_goruck, site-specific extractors, AI function
11.7 Scaling Extraction Across Multiple Sites
The section introduces a batch workflow that repeats AI-assisted product extraction across several pages while recording results and continuing past failures.
- Batch extraction workflow: Trace the four steps repeated for each product: find candidate URLs, have the model choose a page, fetch and clean its HTML, and extract data from the cleaned content. Explain that the workflow records each product’s result and does not stop the whole batch when one product fails.
Why it matters: A batch pipeline must coordinate the extraction steps while keeping one failure from blocking other products. - Curated product test list: Explain that the short, curated product list is sample input chosen for relatively reliable extraction, not part of the orchestration logic, and that a practitioner would replace it with their own list.
Why it matters: Separating example inputs from workflow logic makes clear what to change when testing the batch on your own products.
Key terms: candidate URLs, cleaned HTML, extraction step, batch workflow, curated list, sample input, orchestration logic, companion notebook
11.7.1 One Prompt, Many Sites
11.7.2 Handling Failures and Partial Results
This section explains why product-page extraction can fail or return incomplete data, how to record the outcome, and what a production pipeline needs to handle unreliable results.
- Extraction failures and result statuses: Explain why an extraction may fail or return incomplete data, and distinguish success, error, and a possible partial status in the results.
Why it matters: Clear statuses make it possible to tell usable extractions from failures and incomplete records. - Production failure handling measures: Describe the additional measures proposed for a production extraction pipeline: retries for transient failures, logging every extraction attempt, and a review queue for low-confidence results.
Why it matters: These measures help a pipeline recover from temporary problems and surface results that need human attention.
Key terms: status field, success, error, partial, requests.get(), retry logic, transient failures, logging, review queue, low-confidence results
11.8 Cost and Token Awareness
This section explains how to estimate LLM extraction costs and control them through HTML cleaning and task-appropriate model choices before running a large batch.
- Token-based extraction costs: Explain that LLM APIs charge separately for input and output tokens, with output tokens typically costing more per token. Describe using a token-count and cost estimate before a large batch run rather than judging affordability from a few notebook examples.
Why it matters: A small per-product cost can become significant across hundreds or thousands of pages. - Cleaning and model selection: Explain how aggressive HTML cleaning reduces input tokens and why simpler tasks such as URL ranking and content triage can use a cheaper model while extraction may warrant a more expensive one.
Why it matters: Sending less text and matching model cost to task difficulty can substantially reduce pipeline expense. - Page-level cost comparison: Describe comparing simple, content-heavy, and sparse product pages by measuring raw and cleaned sizes, estimating token counts and costs, and calculating potential savings from a cheaper extraction model. Explain that the choice depends on whether its quality meets the use case's accuracy needs.
Why it matters: Testing different page types makes the cost-versus-quality decision specific to the pages you will process.
Key terms: input tokens, output tokens, tiktoken, token count, cost estimate, aggressive HTML cleaning, URL ranking, content triage, gpt-4o-mini, gpt-4o
11.9 Building Your AI Opportunity Checklist
This section turns the chapter’s examples into a checklist for deciding where AI belongs in a data pipeline, how to justify its cost, and how to handle errors.
- AI task suitability: Explain how input structure, source-specific rules, and the “smart intern test” help identify steps suited to AI: tasks involving messy inputs and human-like judgment are better candidates than fixed operations or tasks requiring precise calculation, specialist expertise, or external access.
Why it matters: These questions help you put AI where it can add value rather than at every pipeline step. - Cost and deterministic alternatives: Describe how to estimate AI’s per-record cost against the value it adds and alternatives such as regex, dictionaries, SQL joins, or ordinary code. Explain why simple transformations, exact lookups, and unjustifiably expensive high-volume decisions should not be sent to an LLM.
Why it matters: A useful AI result may still be the wrong engineering choice if simpler code is reliable or cheaper at scale. - Validation and failure planning: Explain why an AI step needs ways to check its output and a plan for inevitable errors. Describe checks such as schema validation, spot-sampling, known-value comparisons, or confidence scoring, alongside status tracking, review queues, and fallback logic; state that safety-critical outputs must not be used without human review as the sole basis for decisions.
Why it matters: A pipeline must be able to find, review, and recover from AI mistakes rather than silently accepting them.
Key terms: unstructured or semi-structured, smart intern test, site-specific selectors, per-record cost, regex, simple transformations, exact matching, high-volume, low-value decisions, schema checks, spot-sampling
11.10 Lab
This lab has readers run AI-assisted product extraction on a varied batch, then assess field coverage, cost, and how it compares with manual scraping.
11.11 Lab Answers
Where this is used later
- From snippets to a pipeline (chapter 12) · Builds on the chapter’s scraping work to move from snippets toward a pipeline.
- The four agents (chapter 12) · Organizes pipeline responsibilities into agents.
- Agent 4: validate and map (chapter 12) · The validate-and-map step needs to account for the uncertainty of inferred fields.
- (chapter intro) (chapter 12) · URL discovery as an existing piece of the workflow.
- Agent 1: URL discovery (chapter 12) · Implements URL discovery as a pipeline agent
- Agent 2: fetch and clean (chapter 12) · Fetching and cleaning depends on receiving a suitable selected URL.
- Designing the data contract (chapter 12) · The pipeline formalizes its data contract.
- Orchestrating the pipeline (chapter 12) · The chapter turns coordinated steps into pipeline orchestration.
Concepts explained in this chapter
- artificial intelligenceArtificial intelligence is the field of creating computational systems that perform tasks associated with human intelligence, such as reasoning, learning, perception, and decision-making.
- HTML cleaningHTML cleaning is the removal of irrelevant, distracting, or unsafe markup and content from an HTML document before further processing.
- structured extractionAI-powered extraction is the use of an artificial intelligence model to identify and retrieve requested fields from source content based on field descriptions rather than fixed location rules.
- output formatThe response_format parameter is an API request parameter that specifies the desired structure or format of a model response.
- schemaA database schema is the formal organization of a database's tables, fields, relationships, constraints, and related definitions.
- product extractionProduct extraction is the process of extracting product attributes from source content.
- url discoveryURL discovery is the process of finding and selecting a web address for a particular product or other target item.
- unstructured dataUnstructured data is information that lacks a consistently organized schema for its content and fields.
- AI-assisted ranking and filteringAI ranking is the evaluation and ordering of candidate items by an artificial intelligence model according to specified criteria.
- PydanticA Pydantic model is a structured data class defined with Pydantic fields and type annotations that can validate and serialize values.
- site-specific selectorA site-specific selector is a webpage selection rule tailored to the HTML structure of a particular website.
- language modelA large language model is a machine-learning model trained on large text corpora to understand and generate language and related sequences such as code.
- review queueA review queue is an ordered collection of items awaiting inspection or resolution by a reviewer.
- manual extractionManual extraction is the creation and maintenance of explicit rules for retrieving selected fields from source pages.
- data enrichmentEnriched data is information added to source data through inference, generation, or incorporation from external sources.
- data frameA pandas DataFrame is a two-dimensional labeled tabular data structure in the pandas library.
- scrapingWeb scraping is the automated or manual extraction of information from web pages, typically by processing their underlying markup.
- hallucinationAn AI hallucination is a confident but incorrect or misleading output produced by an artificial intelligence model.
- status trackingStatus tracking is the recording and monitoring of the current state of tasks, records, or processing outcomes.
- missing valueA null value is a marker representing the absence or unknown status of a value in a data field.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
