liveBook · Manning

Chapter 5: Using the OpenAI API in Data Workflows

What you will learn in chapter 5 of Learn AI Data Engineering

The chapter shifts AI from an external coding companion to a component inside a data pipeline, using article sentiment analysis as an example of enrichment.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter shifts AI from an external coding companion to a component inside a data pipeline, using article sentiment analysis as an example of enrichment.

Key terms: AI coding companion, data pipeline, enrichment tool, ETL, sentiment analysis, sentiment labels, enrichment, NewsAPI, environment variables, secrets manager

5.1 Initial Setup and Data Extraction

This section introduces NewsAPI as a source of fresh articles and outlines how to retrieve Tesla news for a sentiment-analysis pipeline.

Key terms: NewsAPI, /v2/everything, q, from, to, language, API key, apiKey, JSON, free tier

5.2 Preprocessing Articles

This section prepares extracted news articles for ChatGPT sentiment analysis and suggests a small test of the extraction and preprocessing steps.

Key terms: preprocessing, title, description, content, sentiment analysis, Tesla, five articles, pandas DataFrame

5.3 Using ChatGPT for Sentiment Analysis

Introduces sentiment analysis of prepared articles through the OpenAI Chat Completions API and outlines a gradual approach to making its results more reliable.

Key terms: sentiment analysis, OpenAI Chat Completions API, natural-language prompts, pipeline, crawl-walk-run, plain text prompts, prompt tuning, retry logic, structured output formatting, response_format

5.3.1 Understanding the ChatGPT API and Chat Completions Endpoint

This section introduces the OpenAI Chat Completions API for programmatic sentiment analysis, explains how to verify access, and shows how to control and budget for requests.

Key terms: Chat Completions endpoint, system message, user message, assistant message, sentiment analysis, API key, environment file, OpenAI Python SDK, model, max_tokens

5.3.2 Raw API Response Processing

This section explains how to extract a sentiment result from a raw OpenAI API response, make that result usable in a pipeline, and interpret it cautiously.

Key terms: raw response, content field, metadata, id, model, usage, normalized labels, prompt engineering, response_format, Positive

5.4.1 Normalizing Sentiment Output

Shows how to constrain article sentiment responses to clean labels suitable for structured analysis.

Key terms: sentiment analysis prompt, Positive, Neutral, Negative, df_articles

5.4.2 Logging & Consolidation

This section runs the news-sentiment pipeline end to end, logging each article’s processing and consolidating AI sentiment labels into an enriched DataFrame.

Key terms: preprocessed DataFrame, AI sentiment function, logging, sentiment column, enriched DataFrame, news_api_tsla_full_pipeline.ipynb, NewsAPI, Chat Completions API, API keys

5.5 Lab

This lab has readers adapt a working news-sentiment pipeline, change its output format, and summarize results across more articles.

5.6 Lab Answers

The lab answers show how to change the news query, request numerical sentiment scores, and summarize results after processing more articles.

Where this is used later

Concepts explained in this chapter

← Chapter 4: Using a Coding Companion with Python Chapter 6: AI & Data Quality →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.