liveBook · Manning

NewsAPI

As explained in Learn AI Data Engineering

NewsAPI is a public HTTP service that searches and retrieves articles from news sources and blogs.

NewsAPI is a public HTTP REST service for searching and retrieving live articles from news sources and blogs. It supplies fresh records for monitoring, trend research, and sentiment analysis. section 5.1

Use the /v2/everything endpoint to target articles with:

  • q for keywords or phrases in article titles and content. section 5.1
  • from and to for publication-date ranges. section 5.1
  • language for language selection and sortBy for ordering, such as the latest publication time. section 5.1
  • Raw JSON payloads for preserving article context before later field extraction, validation, or enrichment. section 8.2.1

Daily Tesla extraction

A Python request can query Tesla articles from yesterday through today, using an environment-stored API key, then return the response's articles array. Logging records success or failure, while the resulting records can populate a DataFrame for analysis. section 5.1

News enrichment pipeline

A Disney intelligence workflow can extract the last 24 hours of coverage, retain raw JSON, use AI agents for structured extraction and sentiment, classify business lines and regions, and load enriched data into Postgres. section 8.5

Listing 5.1 Extract Articles
import requests  #A
import pandas as pd  
import logging  
import os
from dotenv import load_dotenv
from datetime import datetime, timedelta  
 
load_dotenv()
 
NEWS_API_KEY = os.getenv("NEWS_API_KEY")  #B
 
# Dynamic date calculation: today minus one day  
today = datetime.now().date()  #C
yesterday = today - timedelta(days=1)  
 
# Function to extract articles from NewsAPI  
def extract_articles(query, from_date=yesterday, api_key=NEWS_API_KEY):  
    url = f'https://newsapi.org/v2/everything?q={query}&from={from_date}&to={today}&apiKey={api_key}'  #D
    response = requests.get(url)  
    
    if response.status_code == 200:  
        articles = response.json().get('articles', [])  #E
        logging.info(f"Successfully extracted {len(articles)} articles.")  
        return articles  
    else:  
        logging.error(f"Failed to fetch articles. Status code: {response.status_code}")  
        return None  
 
# Example use case  
articles = extract_articles('Tesla')  #F
 

Inspect a complete Python extraction pattern using requests, environment configuration, dynamic dates, response checks, JSON parsing, and logging. section 5.1

How Learn AI Data Engineering covers it

How deep it goes
Explains endpoint filters and Python extraction, then stops short of detailed pagination, transformation, and storage implementation.
Examples
Uses Tesla searches, alternative topic queries, and a Disney news pipeline for extraction, categorization, enrichment, and loading.
What it assumes
Assumes familiarity with Python requests, environment variables, JSON responses, and DataFrames.
What you can do after
Apply it via Python requests to retrieve date-filtered articles and stage their JSON for analysis or AI-powered transformation.
How it connects
Connects REST APIs with Python, DataFrames, raw JSON, sentiment analysis, AI agents, and Postgres pipelines.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI Data Engineering. The section links point to the parts of the book it draws on.