Chapter 5: Using the OpenAI API in Data Workflows
The chapter shifts AI from an external coding companion to a component inside a data pipeline, using article sentiment analysis as an example of enrichment.
After this chapter you will be able to
- AI as a pipeline component. Explain the shift from manually asking an AI coding companion for help to embedding an AI model in a data workflow as an enrichment or transformation step.
- Article sentiment enrichment example. Describe the chapter’s example: a small batch of news articles receives sentiment labels from AI, producing results that can flow into downstream data tools. Explain that sentiment analysis illustrates a broader ability to classify, summarize, clean, or reshape data through prompts.
- NewsAPI article selection. Explain why the project uses NewsAPI’s /v2/everything endpoint and how its search and filtering parameters select relevant articles.
- Dynamic article extraction stage. Describe the planned Python extraction function and its role before downstream enrichment.
- Article text preparation. Explain that preprocessing combines each article’s title, description, and content into one clean content field to send to ChatGPT for sentiment analysis.
- Programmable sentiment enrichment. Explain that the prepared articles will be enriched with sentiment analysis by sending natural-language messages to the OpenAI Chat Completions API from a Python workflow. Describe the API as a configurable pipeline component rather than merely a chatbot.
- Gradual prompt refinement. Describe the section’s crawl-walk-run approach: begin with plain-text prompts inside a Python function, then refine the prompts and logic for more reliable results. State that retries and structured output, including JSON schemas through response_format, are more advanced options rather than the starting point.
- Chat Completions message roles. Explain how the Chat Completions endpoint uses messages: a system message guides behavior, a user message supplies the task, and an assistant message contains the model’s response. Relate this exchange to sending article content for sentiment analysis.
- API access and response controls. Describe how to authenticate and try an OpenAI API call using an API key in an environment file and the OpenAI Python SDK version 1.0 or above. Explain that model selects the language model, max_tokens limits response length, and temperature controls randomness.
- Token costs and safeguards. Explain that OpenAI API calls are billed by input and output token usage, so longer prompts, verbose responses, and larger batches increase costs. Describe the section’s safeguards: check current prices, cap output with max_tokens, test on small samples, and monitor usage.
- Sentiment content extraction. Explain that the API returns a structured JSON response with both the sentiment result and metadata, and that extracting and cleaning the content field isolates the result needed for this workflow.
- Pipeline ready sentiment labels. Explain why verbose, inconsistently structured sentiment responses are unsuitable for downstream analytics, and how a more explicit prompt can request only normalized labels such as Positive, Neutral, or Negative.
- Sentiment reliability and validation. Describe ways LLM sentiment labels can be misleading or inconsistent, and explain how explicit instructions about output and method, plus validation before acting, reduce the risk of relying on them.
- Consistent sentiment labels. Apply a prompt that includes an article’s content and asks for only “Positive,” “Neutral,” or “Negative.” Explain why these compact, consistent outputs are more useful for structured analysis than verbose responses, and how to check and refine the prompt if it returns extra text.
- Sentiment logging and consolidation. Describe how the final step loops through the preprocessed DataFrame, applies the AI sentiment function to each article, logs progress, and stores the results in a new sentiment column.
What it builds on
- Interacting with APIs Using AI Coding Companions & Python (chapter 4) · Earlier work used AI coding companions while developing Python code for API interactions.
- Fetching Data from an API (chapter 4) · Fetching data from an API.
- Zero-Shot Prompting (chapter 3) · Introduces prompting with a coding companion before prompts are placed inside a Python function.
- Creating an OpenAI Account (chapter 1) · Creating an OpenAI account.
Section by section
Introduction
The chapter shifts AI from an external coding companion to a component inside a data pipeline, using article sentiment analysis as an example of enrichment.
- AI as a pipeline component: Explain the shift from manually asking an AI coding companion for help to embedding an AI model in a data workflow as an enrichment or transformation step.
Why it matters: This shift is the chapter’s central mental model for using the OpenAI API in data work. - Article sentiment enrichment example: Describe the chapter’s example: a small batch of news articles receives sentiment labels from AI, producing results that can flow into downstream data tools. Explain that sentiment analysis illustrates a broader ability to classify, summarize, clean, or reshape data through prompts.
Why it matters: The article example makes the pipeline integration concrete without limiting AI enrichment to sentiment labels. - Secure API key handling: State that the examples move away from putting API keys directly in scripts and instead assume keys are retrieved from environment variables or a secrets manager, with dotenv used in code examples to load environment settings.
Why it matters: The chapter’s API examples assume credentials are kept outside the scripts that use them.
Key terms: AI coding companion, data pipeline, enrichment tool, ETL, sentiment analysis, sentiment labels, enrichment, NewsAPI, environment variables, secrets manager
5.1 Initial Setup and Data Extraction
This section introduces NewsAPI as a source of fresh articles and outlines how to retrieve Tesla news for a sentiment-analysis pipeline.
- NewsAPI article selection: Explain why the project uses NewsAPI’s /v2/everything endpoint and how its search and filtering parameters select relevant articles.
Why it matters: The sentiment analysis needs fresh, relevant articles rather than an arbitrary set of headlines. - API key and response check: Describe how to verify a NewsAPI key with a browser request and why the example dates must be recent.
Why it matters: Checking access before building the pipeline helps separate API setup problems from later processing problems. - Dynamic article extraction stage: Describe the planned Python extraction function and its role before downstream enrichment.
Why it matters: A repeatable extraction step supplies the articles that later stages will preprocess and analyze.
Key terms: NewsAPI, /v2/everything, q, from, to, language, API key, apiKey, JSON, free tier
5.2 Preprocessing Articles
This section prepares extracted news articles for ChatGPT sentiment analysis and suggests a small test of the extraction and preprocessing steps.
- Article text preparation: Explain that preprocessing combines each article’s title, description, and content into one clean content field to send to ChatGPT for sentiment analysis.
Why it matters: ChatGPT needs the relevant parts of each article together as input for sentiment analysis. - Small pipeline test: Describe testing the extraction and preprocessing functions with the “Tesla” query, limiting the initial run to five articles, and checking for a pandas DataFrame with title, description, and content columns.
Why it matters: A small test checks that the API key works and the setup produces the expected data structure.
Key terms: preprocessing, title, description, content, sentiment analysis, Tesla, five articles, pandas DataFrame
5.3 Using ChatGPT for Sentiment Analysis
Introduces sentiment analysis of prepared articles through the OpenAI Chat Completions API and outlines a gradual approach to making its results more reliable.
- Programmable sentiment enrichment: Explain that the prepared articles will be enriched with sentiment analysis by sending natural-language messages to the OpenAI Chat Completions API from a Python workflow. Describe the API as a configurable pipeline component rather than merely a chatbot.
Why it matters: This frames the API call as a step you can control within a data workflow. - Gradual prompt refinement: Describe the section’s crawl-walk-run approach: begin with plain-text prompts inside a Python function, then refine the prompts and logic for more reliable results. State that retries and structured output, including JSON schemas through response_format, are more advanced options rather than the starting point.
Why it matters: Starting simply makes it easier to improve prompts and assess the reliability of their output before adding complexity.
Key terms: sentiment analysis, OpenAI Chat Completions API, natural-language prompts, pipeline, crawl-walk-run, plain text prompts, prompt tuning, retry logic, structured output formatting, response_format
5.3.1 Understanding the ChatGPT API and Chat Completions Endpoint
This section introduces the OpenAI Chat Completions API for programmatic sentiment analysis, explains how to verify access, and shows how to control and budget for requests.
- Chat Completions message roles: Explain how the Chat Completions endpoint uses messages: a system message guides behavior, a user message supplies the task, and an assistant message contains the model’s response. Relate this exchange to sending article content for sentiment analysis.
Why it matters: These message roles are the basis for sending article text to the model and interpreting its reply. - API access and response controls: Describe how to authenticate and try an OpenAI API call using an API key in an environment file and the OpenAI Python SDK version 1.0 or above. Explain that model selects the language model, max_tokens limits response length, and temperature controls randomness.
Why it matters: A successful small test confirms the connection before AI is embedded in a data pipeline. - Token costs and safeguards: Explain that OpenAI API calls are billed by input and output token usage, so longer prompts, verbose responses, and larger batches increase costs. Describe the section’s safeguards: check current prices, cap output with max_tokens, test on small samples, and monitor usage.
Why it matters: Token discipline helps prevent an affordable experiment from becoming an expensive batch or live-feed job.
Key terms: Chat Completions endpoint, system message, user message, assistant message, sentiment analysis, API key, environment file, OpenAI Python SDK, model, max_tokens
5.3.2 Raw API Response Processing
This section explains how to extract a sentiment result from a raw OpenAI API response, make that result usable in a pipeline, and interpret it cautiously.
- Sentiment content extraction: Explain that the API returns a structured JSON response with both the sentiment result and metadata, and that extracting and cleaning the content field isolates the result needed for this workflow.
Why it matters: A pipeline needs the sentiment result rather than the entire API response. - Pipeline ready sentiment labels: Explain why verbose, inconsistently structured sentiment responses are unsuitable for downstream analytics, and how a more explicit prompt can request only normalized labels such as Positive, Neutral, or Negative.
Why it matters: Consistent categories are easier to store, compare, and use in dashboards or models. - Sentiment reliability and validation: Describe ways LLM sentiment labels can be misleading or inconsistent, and explain how explicit instructions about output and method, plus validation before acting, reduce the risk of relying on them.
Why it matters: A neat-looking sentiment label is not necessarily an objective or reproducible judgment.
Key terms: raw response, content field, metadata, id, model, usage, normalized labels, prompt engineering, response_format, Positive
5.4.1 Normalizing Sentiment Output
Shows how to constrain article sentiment responses to clean labels suitable for structured analysis.
- Consistent sentiment labels: Apply a prompt that includes an article’s content and asks for only “Positive,” “Neutral,” or “Negative.” Explain why these compact, consistent outputs are more useful for structured analysis than verbose responses, and how to check and refine the prompt if it returns extra text.
Why it matters: Clean labels are easier to use in pipelines, dashboards, and downstream machine learning.
Key terms: sentiment analysis prompt, Positive, Neutral, Negative, df_articles
5.4.2 Logging & Consolidation
This section runs the news-sentiment pipeline end to end, logging each article’s processing and consolidating AI sentiment labels into an enriched DataFrame.
- Sentiment logging and consolidation: Describe how the final step loops through the preprocessed DataFrame, applies the AI sentiment function to each article, logs progress, and stores the results in a new sentiment column.
Why it matters: This step turns individual AI predictions into a usable dataset while making processing progress visible. - End-to-end notebook run: Explain what running the companion notebook brings together and what a successful run confirms about the Python environment. State why its news and sentiment results may differ from the chapter’s example.
Why it matters: Running the complete notebook checks that the separate steps and required services work together on live data.
Key terms: preprocessed DataFrame, AI sentiment function, logging, sentiment column, enriched DataFrame, news_api_tsla_full_pipeline.ipynb, NewsAPI, Chat Completions API, API keys
5.5 Lab
This lab has readers adapt a working news-sentiment pipeline, change its output format, and summarize results across more articles.
5.6 Lab Answers
The lab answers show how to change the news query, request numerical sentiment scores, and summarize results after processing more articles.
Where this is used later
- (chapter intro) (chapter 8) · The earlier use of AI to classify News API article sentiment.
- From AI insights to data pipelines (chapter 8) · Introduced using ChatGPT for sentiment analysis.
- Using response_format (chapter 6) · Develops the response_format option mentioned here.
- (chapter intro) (chapter 6) · Introduces the ChatGPT API and Chat Completions endpoint.
- The Transformation Pipeline (chapter 8) · Provides an earlier example of integrating AI enrichment into a data pipeline.
Concepts explained in this chapter
- sentiment analysisSentiment analysis is the computational determination of opinions, emotions, or evaluative polarity expressed in language.
- chat completionThe Chat Completions endpoint is an OpenAI API endpoint that generates assistant responses from conversational messages and related request parameters.
- output formatThe response_format parameter is an API request parameter that specifies the desired structure or format of a model response.
- data frameA pandas DataFrame is a two-dimensional labeled tabular data structure in the pandas library.
- NewsAPINewsAPI is a public HTTP service that searches and retrieves articles from news sources and blogs.
- artificial intelligenceArtificial intelligence is the field of creating computational systems that perform tasks associated with human intelligence, such as reasoning, learning, perception, and decision-making.
- data enrichmentEnriched data is information added to source data through inference, generation, or incorporation from external sources.
- language modelA large language model is a machine-learning model trained on large text corpora to understand and generate language and related sequences such as code.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
