liveBook · Manning

PipelineRecord

As explained in Learn AI Data Engineering

PipelineRecord is a structured model that wraps a product with bookkeeping fields for tracking pipeline progress, confidence, extracted data, validation, and failure state.

PipelineRecord is a validated hand-off object that carries one product through a multi-stage pipeline. It combines product identity with URL, confidence, extracted data, validation results, status, current stage, and failure details. section 12.2

Listing 12.1 The pipeline data contract
from typing import List, Optional
 
from pydantic import BaseModel, Field
 
class ProductInput(BaseModel):  #A
    """What goes into the pipeline: brand + product name."""
    brand_name: str
    product_name: str
 
class ProductExtraction(BaseModel):  #B
    """Raw fields the LLM pulls from a page (from Chapter 11)."""
    product_name: str
    brand_name: str
    description: Optional[str] = None
    price: Optional[str] = None
    weight: Optional[str] = None
    primary_image_url: Optional[str] = None
    category: Optional[str] = None
 
class EnrichedProduct(BaseModel):  #C
    """The validated, normalized record (pipeline Stage 6)."""
    brand_name: str
    product_name: str
    product_url: Optional[str] = None
    description: Optional[str] = None
    price_usd: Optional[float] = None
    weight_value: Optional[float] = None
    weight_unit: Optional[str] = None
    primary_image_url: Optional[str] = None
    category: Optional[str] = None
 
class PipelineRecord(BaseModel):  #D
    """The unit of work that flows through every agent."""
    search_key: str
    status: str = "pending"  #E
    stage: str = "start"  #F
    url: Optional[str] = None
    confidence: Optional[str] = None
    enriched: Optional[EnrichedProduct] = None
    notes: List[str] = Field(default_factory=list)  #G
    error: Optional[str] = None

The data contract defines PipelineRecord alongside ProductInput, ProductExtraction, and EnrichedProduct, separating raw extraction from normalized storage. section 12.2

A PipelineRecord makes each product's progress and outcome explicit:

  • Stores the selected URL and discovery confidence. section 12.2
  • Accumulates state as discovery, fetching, extraction, and validation stages run. section 12.4
  • Records success, failed, or needs_review rather than treating every non-error result as trustworthy. section 12.4
  • Captures the failing stage and error message so batch failures can be diagnosed by pipeline stage. section 12.4
Listing 12.7 The orchestrator: chaining the agents
import time
from typing import Dict, List
 
def run_one(product: ProductInput) -> PipelineRecord:  #A
    """Run a single product through every agent, tracking state."""
    key = f"{product.brand_name} {product.product_name}"
    record = PipelineRecord(search_key=key)
    try:
        record.stage = "discover"  #B
        ranking = discover_url(key)
        if ranking is None:
            record.status = "failed"
            record.error = "no candidate URLs"
            return record
        record.url = ranking.best_url
        record.confidence = ranking.confidence
 
        record.stage = "fetch"  #C
        cleaned = fetch_and_clean(record.url)
 
        record.stage = "extract"  #D
        extraction = extract_product(cleaned)
 
        record.stage = "validate"  #E
        enriched, issues = validate_and_map(
            extraction, record.url, product
        )
        record.enriched = enriched
        record.notes = issues
 
        if issues or ranking.confidence.lower() == "low":  #F
            record.status = "needs_review"
        else:
            record.status = "success"
    except Exception as exc:  #G
        record.status = "failed"
        record.error = f"{type(exc).__name__}: {exc}"
        logger.warning(
            "%s failed at %s: %s", key, record.stage, exc
        )
    return record
 
def run_pipeline(products: List[ProductInput]) -> Dict:  #H
    """Run every product and split results from the review queue."""
    results, review_queue = [], []
    for product in products:
        record = run_one(product)  #I
        results.append(record)
        if record.status == "needs_review":
            review_queue.append(record)
        logger.info("%s -> %s", record.search_key, record.status)
        time.sleep(1)  #J
    return {"results": results, "review_queue": review_queue}

The orchestrator initializes one PipelineRecord, updates its stage before each agent, threads results through it, and assigns the final status. section 12.4

Operational extension

Extend the record with duration_seconds, measure elapsed time in run_one, and include timing in the results DataFrame. A categorization agent may fill a missing category, but its failure should leave the product usable rather than fail the whole record. section 12.8

How Learn AI Data Engineering covers it

How deep it goes
Defines typed bookkeeping fields and staged failure handling, then stops short of specifying every agent's internal implementation.
Examples
Uses URL discovery, fetch, extraction, validation, category completion, timing, and per-stage failure reporting as concrete pipeline operations.
What it assumes
Builds on ProductExtraction and distinguishes it from EnrichedProduct while using typed Pydantic models for hand-offs.
What you can do after
Operate a staged product pipeline, route uncertain records to review, and diagnose batch failures by recorded stage.
How it connects
Connects ProductInput, ProductExtraction, EnrichedProduct, Pydantic, orchestration, validation, and human review.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.