Data cleaning prepares inconsistent or incomplete inputs for reliable downstream use. In a news-embedding workflow, source-specific processing standardizes records before merging them into one dataset. section 11.3.1
A practical workflow can include:
- Normalize each source’s fields and timestamps with dedicated processing functions. section 11.3.1
- Merge records, prioritize content-rich versions, and deduplicate by URL. section 11.3.1
- For credit data, remove columns with excessive missingness, unify categorical variables, and handle outliers with domain-specific caution. section 5.2.3
Cleaning choices depend on the use case: news records need consistent text and metadata for embedding, while credit data preparation feeds feature transformation and model development. section 11.3.1 section 5.2.3
Listing 11.5 A toolkit of source-specific processing functions
import pandas as pd
import numpy as np
# Load all four sources
df_newsapi = pd.read_parquet('newsapi_recent_20251005_0852.parquet')
df_wnn = pd.read_parquet('wnn_20251005_0902.parquet')
df_google = pd.read_parquet('google_rss_20251005_0906.parquet')
df_gdelt = pd.read_parquet('gdelt_2025_08_09_20251005_0858.parquet')
print("="*80)
print("STEP 1: DATA CLEANING & PREPARATION")
print("="*80)
# Source-specific processing functions
def process_newsapi(df):
processed = df[['title', 'description', 'url', 'source_name', 'published_dt']].copy()
processed.columns = ['title', 'content', 'url', 'source', 'published_dt']
processed['data_source'] = 'NewsAPI'
processed['has_content'] = True
processed['published_dt'] = pd.to_datetime(processed['published_dt']).dt.tz_localize(None)
return processed
def process_wnn(df):
processed = df[['title', 'summary', 'url', 'source', 'published_dt']].copy()
processed.columns = ['title', 'content', 'url', 'source', 'published_dt']
processed['data_source'] = 'WNN'
processed['has_content'] = True
processed['published_dt'] = pd.to_datetime(processed['published_dt']).dt.tz_localize(None)
return processed
def process_google(df):
processed = df[['title', 'url', 'source', 'published_dt']].copy()
# Clean Google RSS titles: remove " - Source" suffix
processed['title'] = processed['title'].apply(
lambda x: x.rsplit(' - ', 1)[0] if ' - ' in x else x
)
processed['content'] = None
processed['data_source'] = 'GoogleRSS'
processed['has_content'] = False
processed['published_dt'] = pd.to_datetime(processed['published_dt']).dt.tz_localize(None)
return processed
def process_gdelt(df):
processed = df[['title', 'url', 'domain', 'published_dt']].copy()
processed.columns = ['title', 'url', 'source', 'published_dt']
processed['content'] = None
processed['data_source'] = 'GDELT'
processed['has_content'] = False
processed['published_dt'] = pd.to_datetime(processed['published_dt']).dt.tz_localize(None)
return processed
See source-specific pandas processing functions for a concrete news-data cleaning pattern. section 11.3.1
