liveBook · Manning

Schema

As explained in Learn AI Data Engineering

A database schema is the formal organization of a database's tables, fields, relationships, constraints, and related definitions.

A schema defines the structure that data should have: fields, types, interpretations, and whether values are required or optional. In an AI-assisted pipeline, it is both a contract for downstream code and an instruction set for the model producing structured output. section 11.6.1 section 9.3.2

Use a target schema before extraction, loading, or analysis. Start from the questions the data must answer, then identify one consistent row shape. For pitch events, that can include game context, pitcher and batter, count, pitch type, velocity, and result; unknown or ambiguous values should remain optional rather than being invented. section 9.3.1 section 9.3.2

A practical schema-driven workflow is:

  • Inspect raw records and decide the analytical grain, such as one row per pitch or one row per product. section 9.3.1 section 11.2.1
  • Define named fields, types, requiredness, and concise descriptions in a Pydantic model or comparable structured definition. section 11.6.1
  • Give the model samples and an explicit extraction contract; ask it to surface ambiguity and use null when a value is absent or uncertain. section 9.3.1 section 11.6.2
  • Validate generated SQL or extracted records against the actual data, schema, and business context before relying on them. section 2.0 section 3.1
  • Preserve canonical identifiers so enriched tables can join back to their source records, then handle failures with typed null-filled fallback rows. section 9.3.2

Nested library JSON to typed records

A LibraryBook model specifies library_name, location, title, author, genres, and published_year. Shared library attributes are added to each book payload, while an AI maps nested author and genre data into the defined structure. When the desired shape changes, update the class definition instead of rewriting all parsing logic. section 7.2

Listing 11.4 Defining the product extraction schema (Pydantic)
from pydantic import BaseModel, Field
from typing import Optional
 
class ProductExtraction(BaseModel):
    """Schema for extracting product data from web page content."""
 
    product_name: str = Field(
        description="Full product name as shown on the page"
    )
    brand_name: str = Field(
        description="Manufacturer or brand name"
    )
    description: Optional[str] = Field(
        default=None,
        description="Product description, typically 1-3 sentences"
    )
    price: Optional[str] = Field(
        default=None,
        description="Current retail price including currency symbol"
    )
    weight: Optional[str] = Field(
        default=None,
        description="Product weight with unit (for example, '2.5 lbs' or '1.1 kg')"
    )
    primary_image_url: Optional[str] = Field(
        default=None,
        description="URL of the main product image"
    )
    category: Optional[str] = Field(
        default=None,
        description="Product category (for example, backpack, tent, or sleeping bag)"
    )

This model shows a product schema with required identity fields, optional page-dependent fields, and descriptions that guide extraction as well as validate output. section 11.6.1

Schemas reduce brittle, site-specific parsing. For web pages, describe the product fields wanted instead of encoding where each field appears with CSS selectors or regex. Keep source-formatted values such as prices and weights as strings during extraction when preserving their displayed form matters; normalize them later with deterministic code. section 11.6 section 11.6.1

A schema does not make model output automatically trustworthy. Explicit-presence rules, null for missing values, careful field instructions, and human review limit hallucinated columns, fabricated product details, and invalid SQL. Schema evolution is also an operational event: revise the definition and recheck the pipeline’s assumptions. section 11.6.2 section 2.0 section 7.2

How Learn AI Data Engineering covers it

How deep it goes
Works through Pydantic models, prompts, batch extraction, null handling, and later deterministic normalization rather than database internals.
Examples
Uses nested library JSON, product web pages, news articles, and baseball play-by-play records.
What it assumes
Uses Python, Pydantic, DataFrames, SQL, and LLM prompts while teaching schema design through concrete pipeline tasks.
What you can do after
Build it from scratch by defining typed extraction contracts and validating AI-produced rows or SQL against source data.
How it connects
Connects Pydantic, structured output, SQL DDL, DataFrames, web scraping, data validation, and data cleaning.

Also explained in

Compare how 54 books explain schema →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.