An AI hallucination is a confident but incorrect or misleading output. In data engineering, it can appear as a nonexistent function, an invented dataset column, or SQL that looks valid but fails because of an invalid join or business-rule violation. Plausibility is not evidence: treat generated code and extracted data as a first draft requiring human review. section 2.0
Hallucinations can quietly break automated extraction workflows:
- Invented fields, such as a user_id that does not exist in a log entry, add false data to a schema. section 7.1
- Incorrect regex patterns can compile successfully while extracting dates in the wrong format. section 7.1
- Inconsistent output formats and assumptions drawn from narrow examples make downstream automation unreliable. section 7.1
- Product category and description fields are especially vulnerable when a page does not explicitly provide the value. section 11.6.2
Reduce hallucinations by grounding extraction in an explicit schema and prompt rules: extract only information present in the input, return null when a value cannot be confidently determined, and prohibit guessing. Validate results against known schemas or test data before production. For SQL, request two independent solutions, compare their logic and results, and investigate divergence rather than accepting either answer automatically. section 11.6.2 section 3.4
Schema-bound product extraction
A product-extraction prompt names fields such as product_name, brand_name, price, weight, and category, then requires explicit evidence from cleaned page text. Missing or uncertain values become null rather than fabricated data, protecting downstream pricing analytics from ambiguous guesses. section 11.6.2
Listing 11.5 AI extraction of product fields from cleaned HTML
import openai #A
EXTRACTION_PROMPT = """You are a product data extraction assistant for a data engineering pipeline.
Given the text content of a product web page, extract the following fields accurately:
- product_name: The full product name as displayed on the page
- brand_name: The manufacturer or brand
- description: A concise product description (1-3 sentences)
- price: The current retail price with currency symbol
- weight: The product weight with unit if available
- primary_image_url: The URL of the main product image if found in the text
- category: The product category (backpack, tent, sleeping bag, headlamp, etc.)
Rules:
- Only extract information that is explicitly present in the text
- Use null for any field you cannot find or confidently determine
- Do not guess or fabricate values
- For price, use the current or sale price, not the original price if both are shown
- For weight, include the unit (lbs, oz, kg, g)
- For category, use a simple label based on what the product is""" #B
def extract_product_with_ai( #C
cleaned_text: str,
model: str = "gpt-4o",
) -> ProductExtraction:
"""Extract product fields from cleaned page text using an LLM."""
response = openai.beta.chat.completions.parse( #D
model=model,
messages=[
{"role": "system", "content": EXTRACTION_PROMPT},
{"role": "user", "content": cleaned_text[:8000]}, #E
],
response_format=ProductExtraction, #F
)
return response.choices[0].message.parsed #G
# Example usage
print(f"\nExtracting product data from cleaned HTML...\n")
product = extract_product_with_ai(clean) #H
# Display extraction results
print(f"Extracted Product Data:")
print(f" Name: {product.product_name}")
print(f" Brand: {product.brand_name}")
print(f" Price: {product.price}")
print(f" Weight: {product.weight}")
print(f" Category: {product.category}")
if product.description:
desc_preview = product.description[:100] + "..." if len(product.description) > 100 else product.description
print(f" Description: {desc_preview}")
print(f"\n✓ Product data extracted successfully")
This extraction prompt shows how field definitions and explicit no-guess rules constrain model output from cleaned HTML. section 11.6.2




































