liveBook · Manning

LogExtraction model

As explained in Learn AI Data Engineering

The LogExtraction model is a structured data model representing extracted log type, date, time, and message fields.

LogExtraction is a Pydantic model that gives AI-extracted log data a consistent structure: log type, date, and time. Each field is optional and represented as a string, making the schema suitable for varied log entries. section 7.1

Extending the schema with a message

For `ERROR 2025-06-09 12:34:56 Server failed to respond`, add an optional `message` field, update the extraction prompt, and verify that the output contains `log_type`, `date`, `time`, and `message`. section 7.1

A structured model helps expose common AI-extraction failure modes:

  • Invented fields can appear even when the log contains no corresponding value. section 7.1
  • Inconsistent formatting can make downstream automation unreliable unless the prompt requires explicit structured output. section 7.1
Listing 7.2 AI - Complex Text Processing with Regular Expressions
 
import openai  #A
import os  #A
from dotenv import load_dotenv  #A
from pydantic import BaseModel  #A
from typing import Optional  #A
 
# Load API key from .env file  #B
load_dotenv()  #B
openai.api_key = os.getenv("OPENAI_API_KEY")  #B
 
# Define the data model for extracted output  #C
class LogExtraction(BaseModel):  #C
    log_type: Optional[str]  #C
    date: Optional[str]  #C
    time: Optional[str]  #C
 
# Example log entries  #D
logs = [  #D
    "ERROR 2025-06-09 12:34:56 Server failed to respond",  #D
    "INFO 2025-06-09 12:35:56 User logged in",  #D
    "WARNING 2025-06-09 12:36:56 Disk space low"  #D
]
 
# Prompt for AI to extract log_type, date, and time  #E
row_prompts = [  #E
    "You are a data extraction assistant. Extract the following from the log entry:\n"
    "- log_type: The type of log message (e.g., ERROR, INFO, WARNING)\n"
    "- date: Extract the date in YYYY-MM-DD format\n"
    "- time: Extract the time in HH:MM:SS format\n"
    "Return the result as a JSON object matching the LogExtraction structure."
    for log in logs  #E
]
 
# Process each log entry  #F
for log, prompt in zip(logs, row_prompts):  #F
    try:  #F
        # Make the API call  #F
        completion = openai.beta.chat.completions.parse(  #F
            model="gpt-4o",  #F
            messages=[  #F
                {"role": "system", "content": prompt},  #F
                {"role": "user", "content": log}  #F
            ],  #F
            response_format=LogExtraction  #F
        )
 
        extracted = completion.choices[0].message.parsed.dict()  #F
        print(extracted)  #F
 
    except Exception as e:  #F
        print(f"Error processing log entry: {e}")  #F
 

Inspect the Python implementation, including the Pydantic schema, sample logs, and extraction prompt. section 7.1

How Learn AI Data Engineering covers it

How deep it goes
Defines three optional string fields and extends them with a message field, without detailing validation internals.
Examples
Uses ERROR, INFO, and WARNING log lines containing dates, times, and messages.
What it assumes
Assumes familiarity with Python classes, API-backed extraction prompts, and basic log formats.
What you can do after
Build it from scratch with Pydantic and extend the schema for additional extracted fields.
How it connects
Connects regular expressions, structured data extraction, Pydantic, and prompt-based AI parsing.

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.