liveBook · Manning

Chapter 9: Data Cleaning and Transformation Pipelines in Practice

What you will learn in chapter 9 of Learn AI Data Engineering

The chapter introduces a capstone that brings earlier AI-assisted ETL work into a coordinated pipeline for continuously arriving event data.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduces a capstone that brings earlier AI-assisted ETL work into a coordinated pipeline for continuously arriving event data.

Key terms: event data, data quality checks, transformation logic, lifecycle thinking, Apache Airflow, task based orchestration, event driven, AI powered enrichment, Jupyter notebook

9.1 Data Orchestration

This section introduces orchestration as the way to make data pipelines repeatable and observable, while placing AI inside pipeline tasks rather than using it to design the orchestration.

Key terms: data orchestration, pipeline, tasks, repeatable, observable, Apache Airflow, DAG, sports analytics, individual tasks

9.1.1 Apache Airflow

Introduces Apache Airflow as a way to orchestrate scheduled data pipelines by defining workflows as Python code.

Key terms: workflows as code, Python DAGs, Directed Acyclic Graphs, version-controlled, operators, extensibility, Kubernetes, scheduled workflows

9.1.2 Beyond Scheduling

The section uses an ETL pipeline to show why fixed schedules can produce stale or inaccurate results and why orchestration must account for task dependencies and pipeline state.

Key terms: time based schedules, late arriving data, long running extracts, stale and inaccurate data products, orchestration, dependencies, state of the pipeline, fresh, valid data

9.1.3 Task Framework

This section explains how to turn a pipeline into focused tasks connected by dependencies, then use an orchestrator to run them reliably.

Key terms: task framework, tasks, Extract, Transform, Load, Jupyter, dependencies, directed acyclic graph, DAG, upstream dependencies

9.2 Event Driven Architecture

This section introduces when event-driven architecture fits a data pipeline and how AI can participate as events arrive.

Key terms: event driven architecture, events, streaming, batch extracts, enriching a single message, scoring thousands of events per second

9.2.1 What are events?

This section defines events as time-stamped messages and explains why producers and consumers need an agreed-upon event structure.

Key terms: event, payload, JSON object, producers, consumers, contract, event type, unique identifier, timestamp

9.2.2 Pub/Sub and Beyond

Introduces the publish/subscribe pattern for event-driven pipelines and relates its parts to ETL.

Key terms: producer, event, topic, consumer, publish and subscribe, business defined shape, ETL, continuous extracts, transport and staging layer, transformation and load

9.3 Pipelines in Practice

This section introduces a pipeline that turns natural-language descriptions of individual baseball pitches into structured data for analysis, using event and orchestration concepts.

Key terms: event, pitch, play by play commentary, consistent schema, extraction, validation, transformation, notebook, Apache Airflow, DAG

9.3.1 Inspecting the Data & Inferring the Schema

The section shows how to inspect natural-language pitch events and use AI to propose a reliable, human-reviewed schema before extracting data.

Key terms: pitch event, one row per pitch, target schema, schema, snake_case, required, vagueness, follow-up research, non-deterministic, non coding work

9.3.2 Extracting the basics

This section builds a pitch-by-pitch AI extraction workflow with stable IDs, a typed schema, and a prompt, then explains when to isolate difficult fields in separate extraction passes.

Key terms: Game 7, play_id, play_hash, canonical IDs, Pydantic model, schema, system message, structured output, DataFrame, overloaded extraction prompts

9.3.3 Data Quality Transformations

This section standardizes inconsistently extracted baseball data by matching names and pitch types to canonical reference values before analysis.

Key terms: post-processing, standardize, pitcher_name, canonical reference tables, truth tables, pitcher IDs, pitch type abbreviations, Python dictionaries, Pydantic model, prompt

9.3.4 Advanced Transformations

This section shows how to add AI-generated key-moment labels and excitement scores to structured pitch data, then explains how to prompt for and validate those subjective features.

Key terms: key moment, excitement score, enriched dataset, semantic schema, Pydantic model, system prompt, role assignment, instruction anchoring, JSON object, subjective AI scoring

9.3.5 Analysis

The section uses AI-generated excitement and key-moment features to visualize a baseball game and shows how those features make new analyses possible with familiar data tools.

Key terms: excitement arc, excitement scores, key moments, smoothed trend line, AI engineered features, semantic features, analysis ready table, plain language commentary, full notebook, test notebook

9.4 Lab

The lab asks readers to run the complete Game 7 AI pipeline and use its enriched data to compare pitcher and batter performance.

9.5 Lab Answers

Where this is used later

Concepts explained in this chapter

← Chapter 8: AI and The Data Lifecycle Chapter 10: Introduction to Web Scraping →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.