Synthetic data consists of fictitious records generated to resemble an existing dataset. It expands limited tabular data without adding actual observations, supporting larger machine-learning datasets while preserving the original data’s characteristics. section 2.3.3
Use synthetic data to:
- Address data scarcity by increasing the number of available examples. section 2.3.3
- Improve diversity by strengthening minority classes in imbalanced classification datasets. section 2.3.3
- Generate rare edge cases for testing designed systems. section 2.3.3
- Reduce privacy concerns because generated records are fictitious. section 2.3.3
Listing 2.5 Generating a synthetic dataset
import pandas as pd
from sdv.metadata import SingleTableMetadata #1
from sdv.single_table import TVAESynthesizer #2
data = pd.read_csv("./german_credit_data.csv", index_col=0) #3
data = data.reset_index()
print(data.shape)
print(data.head)
metadata = SingleTableMetadata() #4
metadata.detect_from_dataframe(data) #5
print(metadata.to_dict()) #6
metadata.update_column(column_name="Saving accounts",
sdtype="categorical") #7
synthesizer = TVAESynthesizer(metadata, epochs=10_000) #8
synthesizer.fit(data) #9
synthetic_data = synthesizer.sample(num_rows=10_000) #10
synthetic_data.head()
Generate 10,000 tabular records from a German credit dataset with SDV metadata and a TVAE synthesizer. section 2.3.3
Listing 2.6 Testing a synthetic dataset using adversarial validation
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import roc_auc_score
X = pd.concat([synthetic_data, data]) #1
categorical_columns = [
"Sex", "Housing", "Saving accounts",
"Checking account", "Purpose"]
X_encoded = pd.get_dummies(X, columns=categorical_columns) #2
X= X.drop("index", axis="columns")
y = [0] * len(synthetic_data) + [1] * len(data)
model = RandomForestClassifier()
cv_preds = cross_val_predict(
model, #3
X_encoded, #3
y, #3
cv=5, #3
n_jobs=-1, #3
method="predict_proba" #3
) #3
roc_adv_score = roc_auc_score(y_true=y, y_score=cv_preds[:, 1]) #4
print(f"roc auc adv score: {roc_adv_score:0.3f}")
Test whether synthetic and original records are distinguishable using adversarial validation with a random forest and cross-validated predictions. section 2.3.3
