A joint probability distribution describes how variables occur together, not merely how each variable is distributed alone. It captures dependence across combinations of values: preserving marginal distributions while breaking associations can produce unrealistic, systematically unusual records. section 11.2.2 section 3.1.1
Use the joint distribution to reason about combinations, data sparsity, and synthetic-data realism:
- Under independence, estimate a combination's probability by multiplying its marginal probabilities; with many categorical features, even common individual values yield extremely rare combinations. section 3.1.1
- Preserve associations when generating synthetic data by predicting each feature from previously generated features rather than sampling every feature independently. section 11.2.2 section 11.6
- Estimate low-probability regions for outlier detection with copulas: construct empirical cumulative distributions, form an empirical copula, and estimate each record's tail probability. section 6.4
Baseball features
Synthetic baseball data can match the real marginal distribution of At_bats while failing to preserve its association with RBIs. A left-to-right generator using predictors maintains the joint distribution more effectively; using only the two right-most prior features preserves it less well. section 11.2.2 section 11.6
Listing 11.1 Generating features individually using histograms
import pandas as pd
import numpy as np
from sklearn.datasets import fetch_openml
data = fetch_openml(
'baseball', version=1, parser='auto') #1
df = pd.DataFrame(data.data, columns=data.feature_names)
df['Strikeouts'] = df['Strikeouts'].fillna(df['Strikeouts'].median())
cat_features = [x for x in df.columns
if df[x].nunique() <=10] #2
num_features = [x for x in df.columns
if x not in cat_features]
synth_data = []
for num_feat in num_features: #3
hist = np.histogram(df[num_feat], density=True) #4
bin_centers = [(x+y)/2 for x, y in zip(hist[1][:-1], hist[1][1:])]
p = [x/sum(hist[0]) for x in hist[0]]
vals = np.random.choice(bin_centers, p=p, size=len(df)).astype(int)
vals = [x + (((1.0 * np.random.random()) - 0.5) * df[num_feat].std())
for x in vals]
synth_data.append(vals)
for cat_feat in cat_features: #5
vc = df[cat_feat].value_counts(normalize=True)
vals = np.random.choice(list(vc.index), p=list(vc.values),
size=len(df)) #6
synth_data.append(vals)
synth_df = pd.DataFrame(synth_data).T
synth_df.columns = num_features + cat_features
Inspect a histogram-based generator that samples each numeric feature independently, providing a concrete contrast with joint-distribution modeling. section 11.2.2
