liveBook · Manning

Product extraction

As explained in Learn AI Data Engineering

Product extraction is the process of extracting product attributes from source content.

Product extraction turns cleaned product-page text into structured attributes for an enrichment pipeline. It replaces brittle variant-specific scraping logic with a focused AI-assisted step while keeping the surrounding workflow under human control. section 11.0 section 12.3.4

A practical extraction workflow:

  • Clean the HTML first by removing scripts, navigation, forms, embedded content, review modules, banners, and other non-product regions; this reduces token cost and noise. section 11.5
  • Prompt the model to return product_name, brand_name, description, price, weight, primary_image_url, and category. section 12.3.4
  • Require explicit values only, use null for missing data, preserve price symbols, and retain measurement units rather than guessing. section 12.3.4
  • Retry only genuine API failures with a small budget, then pass the result to deterministic validation and mapping. section 12.3.4 section 12.3.5

Extract, then validate

Given cleaned product-page text, the extractor may return price="$129.99" and weight="1.5 lbs". Deterministic mapping converts these to price_usd=129.99 and normalized weight values. A missing price produces an issue for review; "$0.00" triggers a range check instead of being silently accepted. section 12.3.4 section 12.3.5

Listing 12.5 Agent 3: AI extraction
import openai
 
EXTRACTION_PROMPT = """You are a product data extraction assistant.
Given the text of a product web page, extract these fields:
- product_name, brand_name, description, price, weight,
  primary_image_url, category
Rules:
- Only extract values explicitly present in the text.
- Use null for anything you cannot find; never guess.
- For price, prefer the current/sale price with its symbol.
- For weight, include the unit (lbs, oz, kg, g).
"""  #A
 
@with_retries(max_attempts=2, exceptions=(openai.OpenAIError,))  #B
def extract_product(cleaned_text: str) -> ProductExtraction:  #C
    """Agent: pull structured fields from cleaned text."""
    response = openai.beta.chat.completions.parse(  #D
        model="gpt-4o",
        messages=[
            {"role": "system", "content": EXTRACTION_PROMPT},
            {"role": "user", "content": cleaned_text[:8000]},  #E
        ],
        response_format=ProductExtraction,  #F
    )
    return response.choices[0].message.parsed  #G

Inspect the concrete Python extraction pattern, including the prompt, structured return type, model call, and retry wrapper. section 12.3.4

How Learn AI Data Engineering covers it

How deep it goes
Provides a concrete prompt, structured fields, retry policy, and validation handoff, but defers final trust decisions to orchestration and human review.
Examples
Uses product names, brands, descriptions, prices, weights, image URLs, categories, missing prices, and zero-price validation cases.
What it assumes
Assumes product-page HTML and basic pipeline stages, while teaching the extraction prompt and deterministic checks needed to make outputs usable.
What you can do after
Apply it via Python and an LLM to produce structured product records, then route malformed or incomplete values for review.
How it connects
Connects HTML cleaning, large language models, structured data extraction, data pipelines, validation, normalization, and human review.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.