Structured extraction uses an AI model to turn source content into named, typed fields. Rather than encoding where a value lives with CSS selectors or custom parsers, define the fields wanted in a schema and describe how to find them. The model then interprets variable JSON, HTML, or text and returns a consistent record for downstream data work. section 8.2.2 section 11.2.1 section 11.6
A practical extraction contract names each output field and gives focused instructions. An article contract can require source, title, short summary, and publication date; a product contract can require name, brand, description, price, weight, image URL, and category. Ask for one schema-compliant object, use source content as evidence, and represent unavailable values as null rather than guesses. section 8.2.2 section 12.3.4
Use structured extraction as a controlled pipeline stage:
- Prepare usable input: retain raw article JSON or clean fetched HTML before sending it to the model. section 8.2.2 section 11.10
- Extract target fields in one pass with a schema and concise field-level guidance, instead of per-site selector logic. section 10.2.2 section 11.6
- Map and validate returned values against destination types, missing-field rules, and consistency requirements. section 10.2.2 section 11.10
- Track success or failure, token use, and cost; retry genuine API errors sparingly when extraction calls are expensive. section 11.10 section 12.3.4
Product-page extraction
A manual scraper might read a GORUCK page's h1 and price selectors, returning only title and price and working only for that site. A schema-driven model can instead read cleaned page content and return product name, price, weight, category, and description. This reduces site-specific code, but the resulting fields still need evaluation for completeness and accuracy across a batch. section 11.6.3 section 11.10
Listing 8.2 Extracting Data with AI
import os
import logging
import openai
import pandas as pd
from dotenv import load_dotenv
from pydantic import BaseModel
load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")
class ExtractedArticle(BaseModel): #A
source: str
title: str
short_summary: str
publish_date: str
system_prompt = f""" #B
You are a data extraction agent. For each input article JSON, return a single object matching this schema:
{ExtractedArticle.schema_json(indent=2)}
Use the raw JSON to guide extraction with natural language hints:
- source: use article['source']['name'] when present.
- title: use article['title'].
- short_summary: 1–2 sentences summarizing the article in plain English.
- publish_date: use article['publishedAt'] (ISO-8601 timestamp).
Return exactly one object that matches the schema.
""".strip()
# Sentiment agent (adapted from news_api_tsla_full_pipeline)
def perform_sentiment_analysis(text: str): #C
prompt = (
"Analyze the sentiment of the following text and return a numerical sentiment "
"score from -1 (very negative) to 1 (very positive). Return only the number: "
f"{text}"
)
try:
response = openai.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": prompt}
],
max_tokens=50,
temperature=0.3
)
sentiment_str = response.choices[0].message.content.strip()
return float(sentiment_str)
except Exception as e:
logging.error(f"Error performing sentiment analysis: {e}")
return None
results = []
input_articles = articles # from prior cell
# Limit for quick iteration; adjust/remove as needed
for idx, article in enumerate(input_articles[:5]): #D
try:
completion = openai.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"{article}"}
],
response_format=ExtractedArticle #E
)
parsed = completion.choices[0].message.parsed
if parsed:
item = parsed.dict()
item["sentiment"] = perform_sentiment_analysis(item["short_summary"]) #F
results.append(item)
except Exception as e:
print(f"Error on article {idx}: {e}")
extracted_df = pd.DataFrame(results) #G
print(extracted_df)
The listing shows a Pydantic output contract, field-specific extraction hints, and a request for exactly one object matching the schema. section 8.2.2





