liveBook · Manning

Data transformation

As explained in Learn AI Data Engineering

Data transformation is the process of changing data's structure, representation, or values to meet downstream requirements.

Data transformation changes data values, structure, or representation so downstream systems can use it reliably. In data pipelines, it turns raw records into cleaned, structured, enriched, or analysis-ready outputs. Common work includes standardizing formats, validating fields, deriving categories, combining columns, and reshaping nested records. section 6.3 section 8.1.2

A practical transformation starts with an explicit target: required fields, types, formats, mappings, and rules for invalid or missing data. Traditional Python implementations make each rule traceable through functions, regular expressions, dictionaries, and DataFrame operations. That clarity is useful, but many changing rules can increase code and maintenance effort. section 6.3 section 6.4

Typical data-quality transformations include:

AI-assisted transformation can keep the target schema at the center of the workflow. Define a structured response model, state the expected rules in a prompt, submit records, and write validated returned fields into the dataset. When formats or business rules change, the schema and prompt can be updated instead of rewriting every parsing path. This does not remove the need for clear requirements, controlled inputs, and failure handling. section 6.3 section 6.4 section 7.6

Flattening nested library data

Nested JSON can preserve relationships but remain awkward to query. A library record containing books, nested author details, genre arrays, and shared library attributes can be flattened into one tabular record per book. A Python approach traverses the known JSON structure directly; an AI approach specifies the desired flat schema and extracts matching fields from each nested item. section 7.2

Transformations also create meaning, not only cleanliness. They can generate normalized timestamps, topical or regional labels, engineered features, duplicate-resolution signals, or domain-specific classifications. For subjective labels, define criteria and a schema, preserve structured outputs, and treat model judgment as something to evaluate rather than an automatic fact. section 8.3 section 9.3.4

In ETL, transformation occurs after extraction and before loading into the destination. In ELT, raw data is loaded first and transformed inside the storage environment, often with SQL. Choosing the pattern determines where validation, enrichment, and transformation logic run. section 8.1.2

How Learn AI Data Engineering covers it

How deep it goes
Works through cleaning rules, nested-data flattening, structured AI outputs, and semantic labeling rather than transformation theory.
Examples
Uses messy retail orders, surf-shop records, nested library JSON, news articles, and baseball play-by-play.
What it assumes
Assumes SQL, Python, and AI familiarity helps while keeping the workflow practical and hands-on.
What you can do after
Apply it via Python, pandas, and Chat Completions to clean, flatten, validate, categorize, and enrich records.
How it connects
Connects data transformation with ETL, ELT, data quality, feature engineering, entity resolution, and structured prompting.

Also explained in

Compare how 23 books explain data transformation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.