liveBook · Manning

Structured extraction

As explained in Learn AI Data Engineering

AI-powered extraction is the use of an artificial intelligence model to identify and retrieve requested fields from source content based on field descriptions rather than fixed location rules.

Structured extraction uses an AI model to turn source content into named, typed fields. Rather than encoding where a value lives with CSS selectors or custom parsers, define the fields wanted in a schema and describe how to find them. The model then interprets variable JSON, HTML, or text and returns a consistent record for downstream data work. section 8.2.2 section 11.2.1 section 11.6

A practical extraction contract names each output field and gives focused instructions. An article contract can require source, title, short summary, and publication date; a product contract can require name, brand, description, price, weight, image URL, and category. Ask for one schema-compliant object, use source content as evidence, and represent unavailable values as null rather than guesses. section 8.2.2 section 12.3.4

Use structured extraction as a controlled pipeline stage:

  • Prepare usable input: retain raw article JSON or clean fetched HTML before sending it to the model. section 8.2.2 section 11.10
  • Extract target fields in one pass with a schema and concise field-level guidance, instead of per-site selector logic. section 10.2.2 section 11.6
  • Map and validate returned values against destination types, missing-field rules, and consistency requirements. section 10.2.2 section 11.10
  • Track success or failure, token use, and cost; retry genuine API errors sparingly when extraction calls are expensive. section 11.10 section 12.3.4

Product-page extraction

A manual scraper might read a GORUCK page's h1 and price selectors, returning only title and price and working only for that site. A schema-driven model can instead read cleaned page content and return product name, price, weight, category, and description. This reduces site-specific code, but the resulting fields still need evaluation for completeness and accuracy across a batch. section 11.6.3 section 11.10

Listing 8.2 Extracting Data with AI
import os
import logging
import openai
import pandas as pd
from dotenv import load_dotenv
from pydantic import BaseModel
 
load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")
 
class ExtractedArticle(BaseModel):  #A
    source: str
    title: str
    short_summary: str
    publish_date: str
 
system_prompt = f"""  #B
You are a data extraction agent. For each input article JSON, return a single object matching this schema:
{ExtractedArticle.schema_json(indent=2)}
 
Use the raw JSON to guide extraction with natural language hints:
- source: use article['source']['name'] when present.
- title: use article['title'].
- short_summary: 1–2 sentences summarizing the article in plain English.
- publish_date: use article['publishedAt'] (ISO-8601 timestamp).
 
Return exactly one object that matches the schema.
""".strip()
 
# Sentiment agent (adapted from news_api_tsla_full_pipeline)
def perform_sentiment_analysis(text: str):  #C
    prompt = (
        "Analyze the sentiment of the following text and return a numerical sentiment "
        "score from -1 (very negative) to 1 (very positive). Return only the number: "
        f"{text}"
    )
    try:
        response = openai.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": "You are a helpful assistant."},
                {"role": "user", "content": prompt}
            ],
            max_tokens=50,
            temperature=0.3
        )
        sentiment_str = response.choices[0].message.content.strip()
        return float(sentiment_str)
    except Exception as e:
        logging.error(f"Error performing sentiment analysis: {e}")
        return None
 
results = []
input_articles = articles  # from prior cell
 
# Limit for quick iteration; adjust/remove as needed
for idx, article in enumerate(input_articles[:5]):  #D
    try:
        completion = openai.beta.chat.completions.parse(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": f"{article}"}
            ],
            response_format=ExtractedArticle  #E
        )
        parsed = completion.choices[0].message.parsed
        if parsed:
            item = parsed.dict()
            item["sentiment"] = perform_sentiment_analysis(item["short_summary"])  #F
            results.append(item)
    except Exception as e:
        print(f"Error on article {idx}: {e}")
 
extracted_df = pd.DataFrame(results)  #G
print(extracted_df)
 

The listing shows a Pydantic output contract, field-specific extraction hints, and a request for exactly one object matching the schema. section 8.2.2

How Learn AI Data Engineering covers it

How deep it goes
Works through schemas, prompts, typed outputs, validation, batch evaluation, cost tracking, and narrowly scoped retries.
Examples
Uses news-article JSON, product pages across brands, and baseball play-by-play descriptions as extraction inputs.
What it assumes
Assumes familiarity with Python data workflows and introduces extraction as an alternative to CSS-selector-based scraping.
What you can do after
Apply it via OpenAI and Pydantic to extract schema-shaped fields, evaluate coverage, and operate batch workflows with failure tracking.
How it connects
Connects schema validation, prompt engineering, HTML cleaning, web scraping, data transformation, and Apache Airflow orchestration.

Also explained in

Compare how 6 books explain structured extraction →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.