Part 2 Data Cleaning and Transformation Pipelines with AI
Ask any data engineer where the hours actually go and the answer is rarely the interesting part. It is cleaning, reshaping, and reconciling data that arrives in whatever state the source felt like sending. Part 2 turns AI loose on exactly that work.
Chapter 6 starts with data quality: detecting and fixing the missing values, duplicates, and malformed fields that quietly poison everything downstream. Chapter 7 takes on the transformations that usually mean a browser full of open tabs, complex regex, deeply nested JSON, entity resolution across messy records, and date-time math that respects a real business calendar, and shows how a well-described schema lets the model do the heavy lifting. Chapter 8 zooms out to the full lifecycle, building an extract, transform, and load flow that pulls live news data, enriches it with AI, and lands it somewhere useful.
Chapter 9 is where it becomes a pipeline you could actually run unattended: orchestration with Airflow, event-driven triggers, and the operational concerns that separate a notebook that worked once from a system you trust on a schedule. The throughline is the book's: AI handles the parts it is uniquely good at, and you stay in control of the parts that have to be right.