PipelineRecord is a validated hand-off object that carries one product through a multi-stage pipeline. It combines product identity with URL, confidence, extracted data, validation results, status, current stage, and failure details. section 12.2
Listing 12.1 The pipeline data contract
from typing import List, Optional
from pydantic import BaseModel, Field
class ProductInput(BaseModel): #A
"""What goes into the pipeline: brand + product name."""
brand_name: str
product_name: str
class ProductExtraction(BaseModel): #B
"""Raw fields the LLM pulls from a page (from Chapter 11)."""
product_name: str
brand_name: str
description: Optional[str] = None
price: Optional[str] = None
weight: Optional[str] = None
primary_image_url: Optional[str] = None
category: Optional[str] = None
class EnrichedProduct(BaseModel): #C
"""The validated, normalized record (pipeline Stage 6)."""
brand_name: str
product_name: str
product_url: Optional[str] = None
description: Optional[str] = None
price_usd: Optional[float] = None
weight_value: Optional[float] = None
weight_unit: Optional[str] = None
primary_image_url: Optional[str] = None
category: Optional[str] = None
class PipelineRecord(BaseModel): #D
"""The unit of work that flows through every agent."""
search_key: str
status: str = "pending" #E
stage: str = "start" #F
url: Optional[str] = None
confidence: Optional[str] = None
enriched: Optional[EnrichedProduct] = None
notes: List[str] = Field(default_factory=list) #G
error: Optional[str] = None
The data contract defines PipelineRecord alongside ProductInput, ProductExtraction, and EnrichedProduct, separating raw extraction from normalized storage. section 12.2
A PipelineRecord makes each product's progress and outcome explicit:
- Stores the selected URL and discovery confidence. section 12.2
- Accumulates state as discovery, fetching, extraction, and validation stages run. section 12.4
- Records success, failed, or needs_review rather than treating every non-error result as trustworthy. section 12.4
- Captures the failing stage and error message so batch failures can be diagnosed by pipeline stage. section 12.4
Listing 12.7 The orchestrator: chaining the agents
import time
from typing import Dict, List
def run_one(product: ProductInput) -> PipelineRecord: #A
"""Run a single product through every agent, tracking state."""
key = f"{product.brand_name} {product.product_name}"
record = PipelineRecord(search_key=key)
try:
record.stage = "discover" #B
ranking = discover_url(key)
if ranking is None:
record.status = "failed"
record.error = "no candidate URLs"
return record
record.url = ranking.best_url
record.confidence = ranking.confidence
record.stage = "fetch" #C
cleaned = fetch_and_clean(record.url)
record.stage = "extract" #D
extraction = extract_product(cleaned)
record.stage = "validate" #E
enriched, issues = validate_and_map(
extraction, record.url, product
)
record.enriched = enriched
record.notes = issues
if issues or ranking.confidence.lower() == "low": #F
record.status = "needs_review"
else:
record.status = "success"
except Exception as exc: #G
record.status = "failed"
record.error = f"{type(exc).__name__}: {exc}"
logger.warning(
"%s failed at %s: %s", key, record.stage, exc
)
return record
def run_pipeline(products: List[ProductInput]) -> Dict: #H
"""Run every product and split results from the review queue."""
results, review_queue = [], []
for product in products:
record = run_one(product) #I
results.append(record)
if record.status == "needs_review":
review_queue.append(record)
logger.info("%s -> %s", record.search_key, record.status)
time.sleep(1) #J
return {"results": results, "review_queue": review_queue}
The orchestrator initializes one PipelineRecord, updates its stage before each agent, threads results through it, and assigns the final status. section 12.4
Operational extension
Extend the record with duration_seconds, measure elapsed time in run_one, and include timing in the results DataFrame. A categorization agent may fill a missing category, but its failure should leave the product usable rather than fail the whole record. section 12.8
