liveBook · Manning

Chapter 8: AI and The Data Lifecycle

What you will learn in chapter 8 of Learn AI Data Engineering

The chapter moves from using AI for individual data tasks to integrating it across a News API extract-transform-load pipeline, and introduces orchestration as a way to coordinate that pipeline.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter moves from using AI for individual data tasks to integrating it across a News API extract-transform-load pipeline, and introduces orchestration as a way to coordinate that pipeline.

Key terms: News API, ETL, Extract, Transform, Load, structured fields, normalize, relational database, orchestration, Airflow DAG, retries

8.1 From AI insights to data pipelines

This section shifts from a one-off AI sentiment insight to AI integrated into a sustainable data pipeline, using ETL to frame the engineering work ahead.

Key terms: single insight, full data pipeline, sustainable architecture, ETL, Extract, Transform, Load, data lifecycle, engineering layer

8.1.1 Evolving AI Integration

This subsection explains how useful downstream AI analyses can become shared, guarded features in upstream data pipelines, using an earlier news-sentiment workflow as an example.

Key terms: data lifecycle, feedback loop, operationalizing, upstream pipelines, sentiment, News API, Python, OpenAI API, PostgreSQL, Tableau

8.1.2 Understanding ETL and ELT

This section contrasts ETL and ELT by when transformation happens, then adopts ETL as the model for the chapter’s AI-assisted pipeline.

Key terms: ETL, ELT, Extract, Transform, Load, Extract, Load, Transform, AI-powered extraction, validation, and enrichment, PostgreSQL, cloud data warehouses

8.2 Extracting News Data with AI

This section introduces an AI-assisted approach to extracting news data: keep each article’s full JSON response in one DataFrame row, then use AI to extract the needed fields.

Key terms: News API, JSON response, DataFrame, requests, pandas, API key, .env file, GitHub repository

8.2.1 Extracting the Raw API Payload

This section fetches News API articles as untouched JSON payloads and shows how inspecting them prepares the later AI extraction and transformation steps.

Key terms: News API, extract_articles, raw JSON, DataFrame, article payload, payload size, pagination, batching, publishedAt, source.name

8.2.2 Extracting Data with AI

The section turns raw article JSON into a structured dataset by using a schema-guided AI extraction step and a reusable sentiment-scoring function.

Key terms: ExtractedArticle, pydantic, data contract, response_format, AI harness, perform_sentiment_analysis, max_tokens, temperature, system prompt, user message

8.3 Transforming News Data with AI

This section introduces AI-assisted transformation of extracted news articles and explains why the transformation prompt is kept separate from the code that executes it.

Key terms: enriching, standardizing, normalized timestamps, categorical labels, temporal coverage, structured transformation prompt, intent, execution, schema-conforming results

8.3.1 The Transformation Prompt

This section shows how a schema-driven transformation prompt turns article data into consistently formatted dates and classifications in strict JSON.

Key terms: transformation prompt, schema injection, Pydantic schema, QualityCategorization.schema_json, strict constraints, short_date, publish_date, publish_est, publish_pst, publish_gmt

8.3.2 The AI Data Engineering Code Harness

This section presents a reusable AI data engineering harness that combines a data contract, a precise prompt, and response-handling code to bring language models into structured data workflows.

Key terms: AI data engineering harness, Data Contract, Pydantic classes, Prompt, logic layer, schema serialization, Response Handling Code, operational shell, modularity, reusability

8.3.3 The Transformation Pipeline

This section shows how an AI transformation harness enriches news articles while keeping the resulting dataset structured, validated, and usable downstream.

Key terms: AI Data Engineering Code Harness, structured data contract, QualityCategorization, parse(), response-handling code, temporal and semantic enrichments, short_date, publish_est, topic, region

8.4 Loading News Data with AI

Introduces AI-assisted database loading through a code harness and uses a short SQL exploration exercise to prepare for schema generation.

Key terms: AI Data Engineering Code Harness, glue code, SQL DDL, SQL DML, DataFrame, prompt awareness, schema generation

8.4.1 The Contract and Prompt

This section shows how a data contract and system prompt guide AI to generate a PostgreSQL table definition from an enriched dataset.

Key terms: data contract, pydantic class, system prompt, SQL DDL, CREATE TABLE, response-handling code, ddl string, target database, Postgres

8.4.2 Response Handling

This section shows how the AI Data Engineering Code Harness loads enriched DataFrame rows into PostgreSQL using AI-generated SQL.

Key terms: AI Data Engineering Code Harness, CREATE TABLE, parameterized INSERT, enriched DataFrame, batch-inserted, PostgreSQL, connection settings, access credentials, environment configuration, /setup/postgres_setup.md

8.5 Lab

This lab asks readers to build a Disney news pipeline that extracts recent articles, uses specialized AI agents to structure and enrich them, and loads the results into Postgres.

8.6 Lab Answers

This section points readers to the Chapter 8 lab notebook for full answers and lists the five parts of the lab solution.

Where this is used later

Concepts explained in this chapter

← Chapter 7: AI and Advanced Data Transformations Chapter 9: Data Cleaning and Transformation Pipelines in Practice →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.