Data enrichment adds information not explicitly present in source data. It may infer or generate fields from content, or incorporate external data such as geocoded coordinates or SKU reference mappings. Unlike extraction, enrichment requires judgment rather than simply reading a stated value, so its outputs need closer validation. section 11.2.2 section 11.2
In an AI pipeline, enrichment makes the model a native transformation component: clean source records are sent with explicit instructions, and returned fields are merged for downstream analysis, storage, or reporting. Typical outputs include sentiment labels, standardized summaries, topic and region categories, normalized dates, and inferred use cases. section 5.0 section 8.3 section 11.2.2
Build enrichment as a controlled transformation rather than an unchecked generation step:
- Define a structured data contract for every derived field, including allowed categorical values and required formats. section 8.3.3
- Put the intended business logic in a focused prompt; separate that intent from the code that invokes the model and merges results. section 8.3 section 8.3.3
- Validate model responses against the schema, preserve original values beside derived fields, and use defaults when processing fails. section 8.3.3 section 11.2.2
- Treat deterministic fields differently: use source timestamps as ground truth for conversions and avoid guessing when required input is absent. section 8.3.3
Enriching news articles
A news record can retain its title, summary, and publish date while gaining short_date, EST/PST/GMT publication times, topic, and region. A broader workflow can also assign sentiment and a business-line label, then load the enriched rows into PostgreSQL for routing and analysis. section 8.3.3 section 8.5
Listing 8.3 Transforming Data with AI
import logging
import openai
import pandas as pd
from pydantic import BaseModel #A
class QualityCategorization(BaseModel): #B
short_date: str
publish_est: str
publish_pst: str
publish_gmt: str
topic: str
region: str
qc_system_prompt = f""" #C
You are a data quality and categorization agent. For each input article, return a single object matching this schema:
{QualityCategorization.schema_json(indent=2)}
Instructions:
- short_date: Derive from the input publish_date by dropping time and timezone, format as YYYY-MM-DD.
- publish_est / publish_pst / publish_gmt: Convert the input publish_date to the specified timezone and return ISO-8601 (include timezone offset). Use the original timestamp as ground truth. Do not guess.
- topic: Choose the best label from [Financial, Operations, Product/Technology, Regulatory/Legal, Market/Competition, Executive/Personnel, Strategy/M&A, Customers/Partnerships, Supply Chain/Manufacturing, ESG/Sustainability, Risk/Incidents, Marketing/PR]. If none is perfect, pick the closest and be consistent.
- region: Infer using language cues, source, and content (country/city mentions). Map to one of:
[North America, South America, Europe, Africa, Middle East, Asia, Oceania]. Always use exactly these labels.
- Return strictly valid JSON with exactly these keys and no extra text.
qc_results = [] #D
for idx, row in extracted_df.iterrows(): #E
article_input = {
"source": row.get("source", ""),
"title": row.get("title", ""),
"short_summary": row.get("short_summary", ""),
"publish_date": row.get("publish_date", "")
}
try:
completion = openai.beta.chat.completions.parse( #F
model="gpt-4o",
messages=[
{"role": "system", "content": qc_system_prompt},
{"role": "user", "content": f"{article_input}"}
],
response_format=QualityCategorization
)
parsed = completion.choices[0].message.parsed
if parsed:
qc_results.append(parsed.dict()) #G
else:
qc_results.append({ #H
"short_date": "",
"publish_est": "",
"publish_pst": "",
"publish_gmt": "",
"topic": "",
"region": ""
})
except Exception as e:
logging.error(f"QC error on row {idx}: {e}")
qc_results.append({ #I
"short_date": "",
"publish_est": "",
"publish_pst": "",
"publish_gmt": "",
"topic": "",
"region": ""
})
qc_df = pd.DataFrame(qc_results) #J
enriched_df = pd.concat([extracted_df.reset_index(drop=True), qc_df], axis=1) #K
enriched_df
This implementation pattern combines a Pydantic contract, a schema-aware prompt, parsed model output, validation, fallback handling, and pipeline merging. section 8.3.3

