liveBook · Manning

What you will learn in Learn AI Data Engineering

by David Melillo · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Before You Begin

The chapter introduction frames modern AI as a major technological shift and positions it as a way for skilled professionals to reduce routine work rather than replace human judgment.

1.1 Why AI Matters to Data Engineering · 1.2 Is This Book for You? · 1.2.1 The Many Uses for AI · 1.2.2 The Many Flavors of AI · 1.3 How to Use This Book · 1.3.1 The Main Chapters · 1.3.2 Hands-on Labs · 1.3.3 Chapter Setup Files · 1.4 Setting Up Your Environment · 1.4.1 Installing PostgreSQL and pgAdmin · 1.4.2 Installing Jupyter Lab for Python Work · 1.4.3 Creating an OpenAI Account · 1.5 Being Immediately Effective with AI and Data Engineering

What you learn in chapter 1 →

Chapter 2: Advantages & Disadvantages of Using a Coding Companion

Introduces the productivity benefits of LLM coding companions in data engineering while emphasizing that plausible-looking output requires human validation.

2.1 Mental Model: The Data Engineer and the Coding Companion · 2.2 Advantages of Using an AI/LLM Coding Companion · 2.2.1 Rapid Code Generation for Data Engineering Tasks · 2.3 Disadvantages of Using a Coding Companion · 2.3.1 Introduction to the Pagila Dataset · 2.3.2 Example: Asking a Simple Question · 2.4 Lab · 2.5 Lab Answers

What you learn in chapter 2 →

Chapter 3: Using a Coding Companion with SQL

The chapter introduces using ChatGPT as a coding companion for writing and debugging SQL, with prompting strategies that become more guided as queries grow more complex.

3.1 Zero-Shot Prompting · 3.2 Few-Shot Prompting · 3.3 Chain-of-Thought Prompting · 3.4 Self-Consistency Prompting · 3.5 Tree-of-Thought Prompting · 3.6 Role-Playing, Domain Priming, Prompt Chaining and Beyond · 3.7 Lab · 3.8 Lab Answers

What you learn in chapter 3 →

Chapter 4: Using a Coding Companion with Python

The chapter introduces using an AI coding companion to build and refine Python solutions for repetitive data engineering tasks while keeping the engineer in control.

4.1 Interacting with APIs Using AI Coding Companions & Python · 4.1.1 Fetching Data from an API · 4.1.2 Enhancing API Calls with AI Coding Companions and API Documentation · 4.2 Unnesting Complex JSON Objects with AI Companions & Python · 4.2.1 Simple Example: Flattening a Single Nested Field · 4.2.2 Complex Example: Extracting Deeply Nested & Combined Fields · 4.3 Using AI to Implement Regex Patterns · 4.3.1 Extracting Phone Numbers from Text · 4.3.2 Normalizing Phone Numbers with Regex and AI · 4.3.3 Extracting Number Components into a DataFrame · 4.4 Lab · 4.5 Lab Answers

What you learn in chapter 4 →

Chapter 5: Using the OpenAI API in Data Workflows

The chapter shifts AI from an external coding companion to a component inside a data pipeline, using article sentiment analysis as an example of enrichment.

5.1 Initial Setup and Data Extraction · 5.2 Preprocessing Articles · 5.3 Using ChatGPT for Sentiment Analysis · 5.3.1 Understanding the ChatGPT API and Chat Completions Endpoint · 5.3.2 Raw API Response Processing · 5.4.1 Normalizing Sentiment Output · 5.4.2 Logging & Consolidation · 5.5 Lab · 5.6 Lab Answers

What you learn in chapter 5 →

Chapter 6: AI & Data Quality

The chapter introduces AI as a way to express data quality expectations in prompts rather than stitching together separate rules and tools, then previews a reusable Python approach to validation and cleaning.

6.1 Identifying Data Quality Issues · 6.2 Fixing Data Quality Issues · 6.2.1 Understanding Data Classes · 6.2.2 Using response_format · 6.2.3 Working with Multiple Messages · 6.3 Fixing Structural and Format Issues · 6.4 Lab · 6.5 Lab Answers

What you learn in chapter 6 →

Chapter 7: AI and Advanced Data Transformations

The chapter introduces advanced data transformations as a real-world data engineering need and presents AI as a conversational way to simplify varied transformation work.

7.1 Complex Text Processing with Regular Expressions · 7.2 Handling Hierarchical and Nested Data Structures · 7.3 Entity Resolution · 7.4 Time Series and Date-Time Transformations · 7.5 Lab · 7.6 Lab Answers

What you learn in chapter 7 →

Chapter 8: AI and The Data Lifecycle

The chapter moves from using AI for individual data tasks to integrating it across a News API extract-transform-load pipeline, and introduces orchestration as a way to coordinate that pipeline.

8.1 From AI insights to data pipelines · 8.1.1 Evolving AI Integration · 8.1.2 Understanding ETL and ELT · 8.2 Extracting News Data with AI · 8.2.1 Extracting the Raw API Payload · 8.2.2 Extracting Data with AI · 8.3 Transforming News Data with AI · 8.3.1 The Transformation Prompt · 8.3.2 The AI Data Engineering Code Harness · 8.3.3 The Transformation Pipeline · 8.4 Loading News Data with AI · 8.4.1 The Contract and Prompt · 8.4.2 Response Handling · 8.5 Lab · 8.6 Lab Answers

What you learn in chapter 8 →

Chapter 9: Data Cleaning and Transformation Pipelines in Practice

The chapter introduces a capstone that brings earlier AI-assisted ETL work into a coordinated pipeline for continuously arriving event data.

9.1 Data Orchestration · 9.1.1 Apache Airflow · 9.1.2 Beyond Scheduling · 9.1.3 Task Framework · 9.2 Event Driven Architecture · 9.2.1 What are events? · 9.2.2 Pub/Sub and Beyond · 9.3 Pipelines in Practice · 9.3.1 Inspecting the Data & Inferring the Schema · 9.3.2 Extracting the basics · 9.3.3 Data Quality Transformations · 9.3.4 Advanced Transformations · 9.3.5 Analysis · 9.4 Lab · 9.5 Lab Answers

What you learn in chapter 9 →

Chapter 10: Introduction to Web Scraping

Introduces web scraping through a product-catalog project and explains why manual extraction comes before AI-powered enrichment.

10.1 Why Web Scraping? · 10.1.1 The Gap Between Display and Data · 10.1.2 When Scraping Makes Sense · 10.2 The Product Enrichment Challenge · 10.2.1 From a Simple Product List to a Structured Catalog · 10.2.2 The Enrichment Pipeline · 10.3 Loading the Product Data · 10.4 Finding a Product URL Manually · 10.4.1 Search for the product page · 10.4.2 Evaluate the results and choose a source · 10.4.3 Open the page and inspect what is actually available · 10.5 Web Scraping Fundamentals · 10.5.1 The Classic Scraping Harness: requests plus BeautifulSoup · 10.5.2 Fetching raw HTML with requests · 10.5.3 Parsing and cleaning HTML with BeautifulSoup · 10.6 Examining Real HTML Structure · 10.7 Manual Extraction Across Multiple Sites · 10.8 Lab

What you learn in chapter 10 →

Chapter 11: AI-powered web scraping

This introduction explains why the manual RuckZone scraping approach does not scale and previews an incremental, stage-by-stage use of AI in the enrichment pipeline.

11.1 Where we left off · 11.2 Recognizing AI Opportunities in Data Pipelines · 11.2.1 Extracted Data · 11.2.2 Enriched Data · 11.2.3 Synthetic Data · 11.3 Mapping AI to the Enrichment Pipeline · 11.4 AI-Assisted URL Discovery · 11.4.1 Finding Candidate URLs Programmatically · 11.4.2 Ranking URLs with AI · 11.5 Smarter HTML Cleaning · 11.6 From Manual Selectors to AI Extraction · 11.6.1 Defining What You Want · 11.6.2 Letting AI Do the Extraction · 11.6.3 Manual vs. AI: A Side-by-Side Comparison · 11.7 Scaling Extraction Across Multiple Sites · 11.7.1 One Prompt, Many Sites · 11.7.2 Handling Failures and Partial Results · 11.8 Cost and Token Awareness · 11.9 Building Your AI Opportunity Checklist · 11.10 Lab · 11.11 Lab Answers

What you learn in chapter 11 →

Chapter 12: AI-powered web scraping data pipelines in action

The chapter turns single-product web-scraping snippets into a batch pipeline built from single-purpose agents, with the supporting machinery needed to run it unattended.

12.1 From snippets to a pipeline · 12.1.1 What we mean by "agent" · 12.1.2 Why agents instead of one big function · 12.2 Designing the data contract · 12.3 The four agents · 12.3.1 Shared infrastructure: logging and retries · 12.3.2 Agent 1: URL discovery · 12.3.3 Agent 2: fetch and clean · 12.3.4 Agent 3: AI extraction · 12.3.5 Agent 4: validate and map · 12.4 Orchestrating the pipeline · 12.5 Running the pipeline end to end · 12.6 Working the review queue · 12.7 What you can reuse · 12.8 Lab · 12.9 Lab answers

What you learn in chapter 12 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.