Chapter 6: AI & Data Quality
The chapter introduces AI as a way to express data quality expectations in prompts rather than stitching together separate rules and tools, then previews a reusable Python approach to validation and cleaning.
After this chapter you will be able to
- Traditional and AI quality checks. Explain how traditional data quality work combines rules across SQL, Python, and other tools, and how a conversational prompt can let one model address varied checks within an existing workflow.
- Data quality issue detection. Describe how pandas rules and an AI prompt can identify issues in a small dataset, such as missing values, badly formatted emails, negative numbers, or unwanted duplicates. Explain that pandas requires explicit checks, while a prompt can state broader data quality expectations.
- Validation approach tradeoffs. Explain why hard-coded Python checks offer speed, local execution, no API cost, and control, but depend on knowing the problems in advance. Contrast this with a slower, paid AI approach whose prompts can express changing or ambiguous governance rules and potentially surface unanticipated issues, while vague prompts may lack precision.
- Customer data cleanup example. Describe the duplicate rows for customer id 2 and the all-null optional_note column, and explain how the cleanup example removes them. Contrast writing pandas cleanup logic with describing the desired clean result in a prompt.
- Structured AI cleanup outputs. Explain why the section moves beyond plain prompts and free-form Chat Completions responses: data classes, response formats, and multiple messages are introduced to make AI cleanup output more predictable and programmable.
- CleanedData output contract. Describe how a Pydantic BaseModel defines the expected model response as a CleanedData object, with duplicates as row indexes to remove and drop_columns as column names to drop.
- Response format modes. Distinguish default text responses, JSON mode, and schema-enforced responses by the structure each provides and the parsing each requires.
- Pydantic output contract. Explain how passing a Pydantic data class to response_format defines required output fields and lets the API parse the response into a Python object. Describe the section’s example fields: duplicates as row indexes to remove and drop_columns as column names to drop.
- Free-text parsing risks. Explain why manually parsing plain-text model responses is fragile in production and how structured responses reduce that risk.
- System and user message roles. Explain that the system message sets the model’s data-cleaning role and output expectations, while the user message supplies the data to clean. Describe why separating instructions from data is preferable to putting both in one prompt.
- Structured cleaning response workflow. Describe how a BaseModel data class and the Chat Completions API’s response_format produce a parsed Python object rather than a response that needs manual string parsing. Explain how that structure supports applying AI-detected cleaning results to a DataFrame, such as handling duplicates or mostly null columns.
- Structural and formatting fixes. Describe the dataset problems addressed here and the corresponding fixes: standardize dates and SKU patterns, truncate long text, combine fragmented names, and map product names to standard categories.
- Adaptable AI cleaning pattern. Explain how the Chat Completions API approach reuses the framework from Section 6.2: update the data class and prompt and use response_format to obtain structured outputs, while keeping the core AI-driven code largely the same as quality issues change.
What it builds on
- Understanding the ChatGPT API and Chat Completions Endpoint (chapter 5) · Introduces the ChatGPT API and Chat Completions endpoint.
Section by section
Introduction
The chapter introduces AI as a way to express data quality expectations in prompts rather than stitching together separate rules and tools, then previews a reusable Python approach to validation and cleaning.
- Traditional and AI quality checks: Explain how traditional data quality work combines rules across SQL, Python, and other tools, and how a conversational prompt can let one model address varied checks within an existing workflow.
Why it matters: This contrast explains why the chapter explores AI for data quality. - Chapter validation and cleaning path: Describe the chapter's progression from detecting issues with Python and pandas, through plain-language validations using the ChatGPT Chat Completions API, to a reusable Python function for cleaning and standardizing datasets.
Why it matters: The progression shows how individual checks lead toward a reusable solution.
Key terms: data quality rules, conversational prompt, missing values, invalid formats, contextual anomalies, Python, pandas, ChatGPT Chat Completions API, plain language prompts, reusable, Python-based function
6.1 Identifying Data Quality Issues
This section compares hard-coded pandas checks with natural-language AI prompts for finding data quality issues, emphasizing their different strengths and costs.
- Data quality issue detection: Describe how pandas rules and an AI prompt can identify issues in a small dataset, such as missing values, badly formatted emails, negative numbers, or unwanted duplicates. Explain that pandas requires explicit checks, while a prompt can state broader data quality expectations.
Why it matters: Recognizing how each approach finds problems helps you choose a useful way to inspect data before cleaning it. - Validation approach tradeoffs: Explain why hard-coded Python checks offer speed, local execution, no API cost, and control, but depend on knowing the problems in advance. Contrast this with a slower, paid AI approach whose prompts can express changing or ambiguous governance rules and potentially surface unanticipated issues, while vague prompts may lack precision.
Why it matters: The tradeoff helps you decide when precise programmed rules are enough and when flexible, prompt-based validation is worth trying.
Key terms: pandas, data quality checks, Chat Completions API Endpoint, inconsistencies, hard-coded logic, data governance rules, natural language, flexibility
6.2 Fixing Data Quality Issues
This section uses a small cleanup example to introduce AI-assisted data quality fixes and the structured Chat Completions techniques needed to make their output programmable.
- Customer data cleanup example: Describe the duplicate rows for customer id 2 and the all-null optional_note column, and explain how the cleanup example removes them. Contrast writing pandas cleanup logic with describing the desired clean result in a prompt.
Why it matters: The example makes the move from identifying data quality problems to fixing them concrete. - Structured AI cleanup outputs: Explain why the section moves beyond plain prompts and free-form Chat Completions responses: data classes, response formats, and multiple messages are introduced to make AI cleanup output more predictable and programmable.
Why it matters: A cleanup result is more useful in a data workflow when code can handle its output predictably.
Key terms: duplicate rows, optional_note, pandas, clean, Chat Completions API, data classes, response formats, multiple messages, structured response models
6.2.1 Understanding Data Classes
This section shows how a Pydantic data class defines and validates structured output for an AI-assisted data-cleaning task.
- CleanedData output contract: Describe how a Pydantic BaseModel defines the expected model response as a CleanedData object, with duplicates as row indexes to remove and drop_columns as column names to drop.
Why it matters: A defined output structure makes the model’s cleaning recommendations usable as Python data. - Typed parsing versus strings: Explain that Pydantic can parse and validate the model response into a type-checked CleanedData object, avoiding fragile string parsing, while the OpenAI-powered approach may be slower and incur cost.
Why it matters: Typed responses make AI output clearer and more consistent to integrate into data pipelines.
Key terms: data class, Pydantic, BaseModel, CleanedData, duplicates, drop_columns, parsed, type-checked, string-based parsing, data pipelines
6.2.2 Using response_format
This section shows how response_format makes AI output easier to use in data pipelines by replacing fragile free-text parsing with JSON or schema-enforced responses.
- Response format modes: Distinguish default text responses, JSON mode, and schema-enforced responses by the structure each provides and the parsing each requires.
Why it matters: Choosing the right response format determines how reliably a pipeline can use the model’s output. - Pydantic output contract: Explain how passing a Pydantic data class to response_format defines required output fields and lets the API parse the response into a Python object. Describe the section’s example fields: duplicates as row indexes to remove and drop_columns as column names to drop.
Why it matters: A defined output contract makes AI results easier to validate and use in larger data systems. - Free-text parsing risks: Explain why manually parsing plain-text model responses is fragile in production and how structured responses reduce that risk.
Why it matters: A pipeline can break when a model changes its wording or formatting, even if its underlying answer is similar.
Key terms: response_format, JSON mode, json_object, Default (Text) Mode, Pydantic, BaseModel, duplicates, drop_columns, manual parsing, non-determinism
6.2.3 Working with Multiple Messages
This section separates cleaning instructions from input data in chat messages and combines a data class with structured API output to make AI-assisted DataFrame cleaning easier to reuse.
- System and user message roles: Explain that the system message sets the model’s data-cleaning role and output expectations, while the user message supplies the data to clean. Describe why separating instructions from data is preferable to putting both in one prompt.
Why it matters: Keeping instructions separate from input data can produce more accurate responses as the data becomes more complex. - Structured cleaning response workflow: Describe how a BaseModel data class and the Chat Completions API’s response_format produce a parsed Python object rather than a response that needs manual string parsing. Explain how that structure supports applying AI-detected cleaning results to a DataFrame, such as handling duplicates or mostly null columns.
Why it matters: A predictable Python response makes the cleaning workflow more reliable and reusable. - Testing changes to cleaning contracts: Describe how to test the workflow by extending the data class, changing the prompt to target another quality issue, or swapping datasets, then checking whether the returned structure matches the fields and whether the suggestions remain useful.
Why it matters: Trying changes reveals whether the structured response remains useful beyond the original dataset and cleaning request.
Key terms: system message, user message, data cleaning assistant, BaseModel, data class, response_format, Chat Completions API, parsed Python object, DataFrame, flagged_rows
6.3 Fixing Structural and Format Issues
This section uses a messy retail dataset to compare explicit pandas cleaning steps with a reusable AI-driven approach to structural and formatting problems.
- Structural and formatting fixes: Describe the dataset problems addressed here and the corresponding fixes: standardize dates and SKU patterns, truncate long text, combine fragmented names, and map product names to standard categories.
Why it matters: These inconsistencies can cause errors in downstream systems and analyses. - Explicit pandas cleaning logic: Explain that the traditional pandas approach implements each transformation separately, making its logic explicit and traceable, but that custom functions and additional rules increase maintenance and opportunities for bugs.
Why it matters: The trade-off explains why the section considers an alternative to adding more hand-written rules. - Adaptable AI cleaning pattern: Explain how the Chat Completions API approach reuses the framework from Section 6.2: update the data class and prompt and use response_format to obtain structured outputs, while keeping the core AI-driven code largely the same as quality issues change.
Why it matters: The reusable pattern is the section’s alternative to repeatedly rewriting cleaning logic for new rules.
Key terms: date formats, SKU patterns, text fields, full name, standard categories, pandas, custom function, regular expressions, mapping dictionaries, string manipulation
6.4 Lab
The lab asks readers to clean a surf-shop orders dataset with both pandas and a row-by-row Chat Completions pipeline, then check that the results meet the same data-quality requirements.
6.5 Lab Answers
The lab answers apply pandas cleaning patterns and show why cleaning records one at a time makes AI-assisted data cleaning more reliable.
Where this is used later
- (chapter intro) (chapter 7) · The preceding chapter's focus on using AI to address data quality challenges.
- Data Quality Transformations (chapter 9) · Data quality transformations in a pipeline require decisions about rules and validation.
- Transforming News Data with AI (chapter 8) · Using AI to resolve data quality issues.
Concepts explained in this chapter
- output formatThe response_format parameter is an API request parameter that specifies the desired structure or format of a model response.
- missing valueA null value is a marker representing the absence or unknown status of a value in a data field.
- chat completionThe Chat Completions endpoint is an OpenAI API endpoint that generates assistant responses from conversational messages and related request parameters.
- dataclassA data class is a class whose primary purpose is representing structured data, typically with declarative fields and automatically supplied data-oriented behavior.
- data transformationData transformation is the process of changing data's structure, representation, or values to meet downstream requirements.
- PydanticA Pydantic model is a structured data class defined with Pydantic fields and type annotations that can validate and serialize values.
- data cleaningData cleaning is the process of detecting and correcting, removing, or standardizing inaccurate, incomplete, duplicated, or unsuitable data.
- PythonPython is a general-purpose programming language widely used for automation, data processing, and artificial-intelligence applications.
- data frameA pandas DataFrame is a two-dimensional labeled tabular data structure in the pandas library.
- artificial intelligenceArtificial intelligence is the field of creating computational systems that perform tasks associated with human intelligence, such as reasoning, learning, perception, and decision-making.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
