NewsAPI is a public HTTP REST service for searching and retrieving live articles from news sources and blogs. It supplies fresh records for monitoring, trend research, and sentiment analysis. section 5.1
Use the /v2/everything endpoint to target articles with:
- q for keywords or phrases in article titles and content. section 5.1
- from and to for publication-date ranges. section 5.1
- language for language selection and sortBy for ordering, such as the latest publication time. section 5.1
- Raw JSON payloads for preserving article context before later field extraction, validation, or enrichment. section 8.2.1
Daily Tesla extraction
A Python request can query Tesla articles from yesterday through today, using an environment-stored API key, then return the response's articles array. Logging records success or failure, while the resulting records can populate a DataFrame for analysis. section 5.1
News enrichment pipeline
A Disney intelligence workflow can extract the last 24 hours of coverage, retain raw JSON, use AI agents for structured extraction and sentiment, classify business lines and regions, and load enriched data into Postgres. section 8.5
Listing 5.1 Extract Articles
import requests #A
import pandas as pd
import logging
import os
from dotenv import load_dotenv
from datetime import datetime, timedelta
load_dotenv()
NEWS_API_KEY = os.getenv("NEWS_API_KEY") #B
# Dynamic date calculation: today minus one day
today = datetime.now().date() #C
yesterday = today - timedelta(days=1)
# Function to extract articles from NewsAPI
def extract_articles(query, from_date=yesterday, api_key=NEWS_API_KEY):
url = f'https://newsapi.org/v2/everything?q={query}&from={from_date}&to={today}&apiKey={api_key}' #D
response = requests.get(url)
if response.status_code == 200:
articles = response.json().get('articles', []) #E
logging.info(f"Successfully extracted {len(articles)} articles.")
return articles
else:
logging.error(f"Failed to fetch articles. Status code: {response.status_code}")
return None
# Example use case
articles = extract_articles('Tesla') #F
Inspect a complete Python extraction pattern using requests, environment configuration, dynamic dates, response checks, JSON parsing, and logging. section 5.1
