Status tracking records a pipeline record’s current stage and outcome while work moves through discovery, fetching, extraction, validation, and persistence. Recording the stage before each agent runs makes failures diagnosable instead of merely countable. section 12.4
Use explicit states to separate trusted output from work requiring intervention:
- Set stage to the active operation, such as discover, fetch, extract, or validate. section 12.4
- Mark a record success only when its output passes validation and is ready to persist. section 12.6
- Mark a record failed when processing cannot continue, and retain the error and stage. section 12.4
- Mark a record needs_review when validation issues or low confidence make human inspection necessary. section 12.6
A safe pipeline handoff
A batch of 450 products can reveal whether failures cluster in fetch or extraction. After processing, show the review queue with each record’s confidence, URL, and issues, then write only success records to enriched_products.csv. Failed and needs_review records remain outside the catalog until resolved. section 12.4 section 12.6
Listing 12.7 The orchestrator: chaining the agents
import time
from typing import Dict, List
def run_one(product: ProductInput) -> PipelineRecord: #A
"""Run a single product through every agent, tracking state."""
key = f"{product.brand_name} {product.product_name}"
record = PipelineRecord(search_key=key)
try:
record.stage = "discover" #B
ranking = discover_url(key)
if ranking is None:
record.status = "failed"
record.error = "no candidate URLs"
return record
record.url = ranking.best_url
record.confidence = ranking.confidence
record.stage = "fetch" #C
cleaned = fetch_and_clean(record.url)
record.stage = "extract" #D
extraction = extract_product(cleaned)
record.stage = "validate" #E
enriched, issues = validate_and_map(
extraction, record.url, product
)
record.enriched = enriched
record.notes = issues
if issues or ranking.confidence.lower() == "low": #F
record.status = "needs_review"
else:
record.status = "success"
except Exception as exc: #G
record.status = "failed"
record.error = f"{type(exc).__name__}: {exc}"
logger.warning(
"%s failed at %s: %s", key, record.stage, exc
)
return record
def run_pipeline(products: List[ProductInput]) -> Dict: #H
"""Run every product and split results from the review queue."""
results, review_queue = [], []
for product in products:
record = run_one(product) #I
results.append(record)
if record.status == "needs_review":
review_queue.append(record)
logger.info("%s -> %s", record.search_key, record.status)
time.sleep(1) #J
return {"results": results, "review_queue": review_queue}
Inspect the orchestrator pattern for threading state through a PipelineRecord, recording stages, and assigning outcomes. section 12.4
