liveBook · Manning

Chapter 7: AI and Advanced Data Transformations

What you will learn in chapter 7 of Learn AI Data Engineering

The chapter introduces advanced data transformations as a real-world data engineering need and presents AI as a conversational way to simplify varied transformation work.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduces advanced data transformations as a real-world data engineering need and presents AI as a conversational way to simplify varied transformation work.

Key terms: data quality, advanced data transformations, context switching, conversational interface, one size fits all

7.1 Complex Text Processing with Regular Expressions

This section shows how to turn messy log entries into structured data using either a precise regular expression or AI-assisted extraction, and why AI results must be validated.

Key terms: log file, structured fields, query-ready data, regex, capture groups, log level, \s, AI-driven approach, conversational prompts, LogExtraction model

7.2 Handling Hierarchical and Nested Data Structures

This section compares hand-coded Python flattening with AI-assisted mapping of nested library JSON into structured book records.

Key terms: nested JSON, flatten, DataFrame, author, genres, Pydantic, data class, target schema, payload, shared attributes

7.3 Entity Resolution

This section compares hand-coded and AI-based entity resolution, then shows why reliable AI matching depends on carefully specified and calibrated prompts.

Key terms: entity resolution, real-world entity, shared signals, string normalization, similarity scoring, heuristic rules, field weighting, fingerprints, AI-based approach, prompt

7.4 Time Series and Date-Time Transformations

The section transforms accounts-receivable transactions into reporting-ready dates, due dates, and balance measures, then compares traditional Python with an AI-assisted approach.

Key terms: transaction_date, PST, EST, GST, fiscal quarter, NET30, NET60, business days, account_totals, total outstanding balance

7.5 Lab

The lab asks readers to use Python and AI to clean, merge, and analyze three messy customer and transaction datasets.

7.6 Lab Answers

The lab answers show how AI can handle varied data transformations while making changing formats and business rules easier to accommodate.

Where this is used later

Concepts explained in this chapter

← Chapter 6: AI & Data Quality Chapter 8: AI and The Data Lifecycle →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.