liveBook · Manning

Pydantic

As explained in Learn AI Data Engineering

A Pydantic model is a structured data class defined with Pydantic fields and type annotations that can validate and serialize values.

Pydantic defines structured Python data with classes, fields, and type annotations. A model such as a BaseModel subclass acts as a schema-like contract: it describes expected values and validates whether returned data fits that shape. In AI-assisted pipelines, this replaces fragile interpretation of free-form text with a typed object that downstream code can use directly. section 6.2.1 section 6.2.2

from pydantic import BaseModel
from typing import List
 
class CleanedData(BaseModel):
    duplicates: List[int]
    drop_columns: List[str]

A minimal model specifies two cleanup decisions: duplicate row indexes and column names to remove. The field types make the intended output explicit. section 6.2.1

Use a Pydantic model as the response contract when an LLM must produce data for automation. Passing the model through response_format directs the API toward the required JSON structure and parses the result into Python, avoiding regex, string slicing, and manual post-processing. JSON mode is lighter-weight, but can still produce malformed output and does not impose the same strict field structure. section 6.2.2

Design models around the pipeline decision or record you need next:

  • Define required fields for values that must be present, and optional fields where extraction may be unavailable. section 9.3.2
  • Include narrowly scoped decision fields when classifying or ranking, such as a chosen URL, confidence, and reasoning. section 11.4.2
  • Update the class when a nested-data target schema evolves instead of rewriting custom extraction logic. section 7.2
  • Pair the schema with explicit prompt instructions for complex or subjective fields, including rules and examples where needed. section 9.3.4

Extracting baseball play data

A PlayByPlayExtraction model defines typed fields for inning, players, counts, pitch details, and results. A loop sends each commentary row to the parsing API with that model as response_format, converts the parsed object to a dictionary, then builds a DataFrame and merges it back using canonical IDs. If a call fails, the workflow appends a record with fields set to None rather than stopping the batch. section 9.3.2

Constraining an enrichment agent

A categorization agent can use a Pydantic response model to constrain an LLM to one category from a fixed product list. Run it only when extraction leaves the category missing, and treat its failure as nonfatal so the product pipeline can complete with an absent category. section 12.8

Pydantic improves clarity, consistency, debuggability, and reuse where AI output feeds larger data systems. It does not make generated judgments automatically correct: prompts, validation rules, logging, error handling, and review remain important, especially for ambiguous extraction or subjective scoring. section 6.2.1 section 9.3.4

How Learn AI Data Engineering covers it

How deep it goes
Explains BaseModel contracts through complete response-format and batch-extraction patterns, while deferring deeper library internals.
Examples
Uses cleanup decisions, library-book JSON, baseball play extraction, URL ranking, and product categorization.
What it assumes
Teaches Pydantic from zero while assuming practical Python classes, type annotations, and pipeline work.
What you can do after
Apply it via OpenAI parsing APIs to obtain validated Python objects and preserve batch progress through failures.
How it connects
Connects Pydantic to response_format, JSON, nested data, DataFrames, structured prompting, and Airflow pipelines.

Also explained in

Compare how 6 books explain Pydantic →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.