liveBook · Manning

Status tracking

As explained in Learn AI Data Engineering

Status tracking is the recording and monitoring of the current state of tasks, records, or processing outcomes.

Status tracking records a pipeline record’s current stage and outcome while work moves through discovery, fetching, extraction, validation, and persistence. Recording the stage before each agent runs makes failures diagnosable instead of merely countable. section 12.4

Use explicit states to separate trusted output from work requiring intervention:

  • Set stage to the active operation, such as discover, fetch, extract, or validate. section 12.4
  • Mark a record success only when its output passes validation and is ready to persist. section 12.6
  • Mark a record failed when processing cannot continue, and retain the error and stage. section 12.4
  • Mark a record needs_review when validation issues or low confidence make human inspection necessary. section 12.6

A safe pipeline handoff

A batch of 450 products can reveal whether failures cluster in fetch or extraction. After processing, show the review queue with each record’s confidence, URL, and issues, then write only success records to enriched_products.csv. Failed and needs_review records remain outside the catalog until resolved. section 12.4 section 12.6

Listing 12.7 The orchestrator: chaining the agents
import time
from typing import Dict, List
 
def run_one(product: ProductInput) -> PipelineRecord:  #A
    """Run a single product through every agent, tracking state."""
    key = f"{product.brand_name} {product.product_name}"
    record = PipelineRecord(search_key=key)
    try:
        record.stage = "discover"  #B
        ranking = discover_url(key)
        if ranking is None:
            record.status = "failed"
            record.error = "no candidate URLs"
            return record
        record.url = ranking.best_url
        record.confidence = ranking.confidence
 
        record.stage = "fetch"  #C
        cleaned = fetch_and_clean(record.url)
 
        record.stage = "extract"  #D
        extraction = extract_product(cleaned)
 
        record.stage = "validate"  #E
        enriched, issues = validate_and_map(
            extraction, record.url, product
        )
        record.enriched = enriched
        record.notes = issues
 
        if issues or ranking.confidence.lower() == "low":  #F
            record.status = "needs_review"
        else:
            record.status = "success"
    except Exception as exc:  #G
        record.status = "failed"
        record.error = f"{type(exc).__name__}: {exc}"
        logger.warning(
            "%s failed at %s: %s", key, record.stage, exc
        )
    return record
 
def run_pipeline(products: List[ProductInput]) -> Dict:  #H
    """Run every product and split results from the review queue."""
    results, review_queue = [], []
    for product in products:
        record = run_one(product)  #I
        results.append(record)
        if record.status == "needs_review":
            review_queue.append(record)
        logger.info("%s -> %s", record.search_key, record.status)
        time.sleep(1)  #J
    return {"results": results, "review_queue": review_queue}

Inspect the orchestrator pattern for threading state through a PipelineRecord, recording stages, and assigning outcomes. section 12.4

How Learn AI Data Engineering covers it

How deep it goes
Implements stage fields, error capture, three-way outcomes, review queues, and output filtering without covering general monitoring systems.
Examples
Uses PipelineRecord stages, success/needs_review/failed states, review-queue displays, and filtered CSV catalogs.
What it assumes
Assumes a staged processing pipeline and validation checks, while treating error handling, review routing, and confidence thresholds as design decisions.
What you can do after
Diagnose/debug with it by locating failure stages, routing uncertain records to people, and preventing untrusted output from reaching the database.
How it connects
Connects pipeline orchestration with validation, confidence scoring, review queues, fallback logic, and human review.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.