liveBook · Manning

Chapter 6: AI & Data Quality

What you will learn in chapter 6 of Learn AI Data Engineering

The chapter introduces AI as a way to express data quality expectations in prompts rather than stitching together separate rules and tools, then previews a reusable Python approach to validation and cleaning.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduces AI as a way to express data quality expectations in prompts rather than stitching together separate rules and tools, then previews a reusable Python approach to validation and cleaning.

Key terms: data quality rules, conversational prompt, missing values, invalid formats, contextual anomalies, Python, pandas, ChatGPT Chat Completions API, plain language prompts, reusable, Python-based function

6.1 Identifying Data Quality Issues

This section compares hard-coded pandas checks with natural-language AI prompts for finding data quality issues, emphasizing their different strengths and costs.

Key terms: pandas, data quality checks, Chat Completions API Endpoint, inconsistencies, hard-coded logic, data governance rules, natural language, flexibility

6.2 Fixing Data Quality Issues

This section uses a small cleanup example to introduce AI-assisted data quality fixes and the structured Chat Completions techniques needed to make their output programmable.

Key terms: duplicate rows, optional_note, pandas, clean, Chat Completions API, data classes, response formats, multiple messages, structured response models

6.2.1 Understanding Data Classes

This section shows how a Pydantic data class defines and validates structured output for an AI-assisted data-cleaning task.

Key terms: data class, Pydantic, BaseModel, CleanedData, duplicates, drop_columns, parsed, type-checked, string-based parsing, data pipelines

6.2.2 Using response_format

This section shows how response_format makes AI output easier to use in data pipelines by replacing fragile free-text parsing with JSON or schema-enforced responses.

Key terms: response_format, JSON mode, json_object, Default (Text) Mode, Pydantic, BaseModel, duplicates, drop_columns, manual parsing, non-determinism

6.2.3 Working with Multiple Messages

This section separates cleaning instructions from input data in chat messages and combines a data class with structured API output to make AI-assisted DataFrame cleaning easier to reuse.

Key terms: system message, user message, data cleaning assistant, BaseModel, data class, response_format, Chat Completions API, parsed Python object, DataFrame, flagged_rows

6.3 Fixing Structural and Format Issues

This section uses a messy retail dataset to compare explicit pandas cleaning steps with a reusable AI-driven approach to structural and formatting problems.

Key terms: date formats, SKU patterns, text fields, full name, standard categories, pandas, custom function, regular expressions, mapping dictionaries, string manipulation

6.4 Lab

The lab asks readers to clean a surf-shop orders dataset with both pandas and a row-by-row Chat Completions pipeline, then check that the results meet the same data-quality requirements.

6.5 Lab Answers

The lab answers apply pandas cleaning patterns and show why cleaning records one at a time makes AI-assisted data cleaning more reliable.

Where this is used later

Concepts explained in this chapter

← Chapter 5: Using the OpenAI API in Data Workflows Chapter 7: AI and Advanced Data Transformations →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.