liveBook · Manning

Data enrichment

As explained in Learn AI Data Engineering

Enriched data is information added to source data through inference, generation, or incorporation from external sources.

Data enrichment adds information not explicitly present in source data. It may infer or generate fields from content, or incorporate external data such as geocoded coordinates or SKU reference mappings. Unlike extraction, enrichment requires judgment rather than simply reading a stated value, so its outputs need closer validation. section 11.2.2 section 11.2

In an AI pipeline, enrichment makes the model a native transformation component: clean source records are sent with explicit instructions, and returned fields are merged for downstream analysis, storage, or reporting. Typical outputs include sentiment labels, standardized summaries, topic and region categories, normalized dates, and inferred use cases. section 5.0 section 8.3 section 11.2.2

Build enrichment as a controlled transformation rather than an unchecked generation step:

  • Define a structured data contract for every derived field, including allowed categorical values and required formats. section 8.3.3
  • Put the intended business logic in a focused prompt; separate that intent from the code that invokes the model and merges results. section 8.3 section 8.3.3
  • Validate model responses against the schema, preserve original values beside derived fields, and use defaults when processing fails. section 8.3.3 section 11.2.2
  • Treat deterministic fields differently: use source timestamps as ground truth for conversions and avoid guessing when required input is absent. section 8.3.3

Enriching news articles

A news record can retain its title, summary, and publish date while gaining short_date, EST/PST/GMT publication times, topic, and region. A broader workflow can also assign sentiment and a business-line label, then load the enriched rows into PostgreSQL for routing and analysis. section 8.3.3 section 8.5

Listing 8.3 Transforming Data with AI
import logging
import openai
import pandas as pd
from pydantic import BaseModel  #A
 
class QualityCategorization(BaseModel):  #B
    short_date: str
    publish_est: str
    publish_pst: str
    publish_gmt: str
    topic: str
    region: str
 
qc_system_prompt = f"""  #C
You are a data quality and categorization agent. For each input article, return a single object matching this schema:
{QualityCategorization.schema_json(indent=2)}
 
Instructions:
- short_date: Derive from the input publish_date by dropping time and timezone, format as YYYY-MM-DD.
- publish_est / publish_pst / publish_gmt: Convert the input publish_date to the specified timezone and return ISO-8601 (include timezone offset). Use the original timestamp as ground truth. Do not guess.
- topic: Choose the best label from [Financial, Operations, Product/Technology, Regulatory/Legal, Market/Competition, Executive/Personnel, Strategy/M&A, Customers/Partnerships, Supply Chain/Manufacturing, ESG/Sustainability, Risk/Incidents, Marketing/PR]. If none is perfect, pick the closest and be consistent.
- region: Infer using language cues, source, and content (country/city mentions). Map to one of:
  [North America, South America, Europe, Africa, Middle East, Asia, Oceania]. Always use exactly these labels.
- Return strictly valid JSON with exactly these keys and no extra text.
 
qc_results = []  #D
 
for idx, row in extracted_df.iterrows():  #E
    article_input = {
        "source": row.get("source", ""),
        "title": row.get("title", ""),
        "short_summary": row.get("short_summary", ""),
        "publish_date": row.get("publish_date", "")
    }
    try:
        completion = openai.beta.chat.completions.parse(  #F
            model="gpt-4o",
            messages=[
                {"role": "system", "content": qc_system_prompt},
                {"role": "user", "content": f"{article_input}"}
            ],
            response_format=QualityCategorization
        )
        parsed = completion.choices[0].message.parsed
        if parsed:
            qc_results.append(parsed.dict())  #G
        else:
            qc_results.append({  #H
                "short_date": "",
                "publish_est": "",
                "publish_pst": "",
                "publish_gmt": "",
                "topic": "",
                "region": ""
            })
    except Exception as e:
        logging.error(f"QC error on row {idx}: {e}")
        qc_results.append({  #I
            "short_date": "",
            "publish_est": "",
            "publish_pst": "",
            "publish_gmt": "",
            "topic": "",
            "region": ""
        })
qc_df = pd.DataFrame(qc_results)  #J
enriched_df = pd.concat([extracted_df.reset_index(drop=True), qc_df], axis=1)  #K
enriched_df
 

This implementation pattern combines a Pydantic contract, a schema-aware prompt, parsed model output, validation, fallback handling, and pipeline merging. section 8.3.3

How Learn AI Data Engineering covers it

How deep it goes
Works through schema contracts, prompts, parsed responses, validation, defaults, and pipeline merging rather than treating enrichment as automatic.
Examples
Uses news articles enriched with sentiment, topic, region, timezone-normalized dates, and Disney business-line labels.
What it assumes
Assumes helpful familiarity with SQL, Python, and AI concepts while emphasizing practical pipeline implementation.
What you can do after
Build it from scratch with structured prompts, validated model outputs, graceful fallbacks, and downstream-ready enriched rows.
How it connects
Connects extraction, data quality, feature engineering, sentiment analysis, schema validation, and ETL pipelines.

Also explained in

Compare how 2 books explain data enrichment →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.