Product extraction turns cleaned product-page text into structured attributes for an enrichment pipeline. It replaces brittle variant-specific scraping logic with a focused AI-assisted step while keeping the surrounding workflow under human control. section 11.0 section 12.3.4
A practical extraction workflow:
- Clean the HTML first by removing scripts, navigation, forms, embedded content, review modules, banners, and other non-product regions; this reduces token cost and noise. section 11.5
- Prompt the model to return product_name, brand_name, description, price, weight, primary_image_url, and category. section 12.3.4
- Require explicit values only, use null for missing data, preserve price symbols, and retain measurement units rather than guessing. section 12.3.4
- Retry only genuine API failures with a small budget, then pass the result to deterministic validation and mapping. section 12.3.4 section 12.3.5
Extract, then validate
Given cleaned product-page text, the extractor may return price="$129.99" and weight="1.5 lbs". Deterministic mapping converts these to price_usd=129.99 and normalized weight values. A missing price produces an issue for review; "$0.00" triggers a range check instead of being silently accepted. section 12.3.4 section 12.3.5
Listing 12.5 Agent 3: AI extraction
import openai
EXTRACTION_PROMPT = """You are a product data extraction assistant.
Given the text of a product web page, extract these fields:
- product_name, brand_name, description, price, weight,
primary_image_url, category
Rules:
- Only extract values explicitly present in the text.
- Use null for anything you cannot find; never guess.
- For price, prefer the current/sale price with its symbol.
- For weight, include the unit (lbs, oz, kg, g).
""" #A
@with_retries(max_attempts=2, exceptions=(openai.OpenAIError,)) #B
def extract_product(cleaned_text: str) -> ProductExtraction: #C
"""Agent: pull structured fields from cleaned text."""
response = openai.beta.chat.completions.parse( #D
model="gpt-4o",
messages=[
{"role": "system", "content": EXTRACTION_PROMPT},
{"role": "user", "content": cleaned_text[:8000]}, #E
],
response_format=ProductExtraction, #F
)
return response.choices[0].message.parsed #G
Inspect the concrete Python extraction pattern, including the prompt, structured return type, model call, and retry wrapper. section 12.3.4
