A real versus fake outlier detector creates synthetic records, trains a binary classifier to distinguish real from artificial data, and treats real records classified as artificial as outliers. It applies the predictive-model principle that poorly predicted values may be anomalous. section 3.2.5
A practical workflow includes:
- Generate synthetic values for each feature, here by sampling normal distributions fitted to the real columns. section 3.2.5
- Combine real and synthetic records, label their origin, and train a classifier such as a decision tree. section 3.2.5
- Tune both synthetic-data generation and classifier settings, then inspect false positives and false negatives rather than treating every flag as meaningful. section 3.2.5
Breast-cancer records
Using 569 real records and 569 normally generated synthetic records, a tuned decision tree identified 544 real records as real and flagged 25 as fake; it classified 561 synthetic records as fake. section 3.2.5
Listing 3.7 Predicting real vs. synthetic data
import pandas as pd
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import confusion_matrix
from sklearn.tree import export_text
np.random.seed(0)
data = load_breast_cancer() #1
data_df = pd.DataFrame(data.data, columns=data.feature_names)
data_df['Real'] = True
synth_df = pd.DataFrame() #2
for col_name in data_df.columns:
mean = data_df[col_name].mean()
stddev = data_df[col_name].std()
synth_df[col_name] = np.random.normal(
loc=mean, scale=stddev, size=len(data_df))
synth_df['Real'] = False
train_df = pd.concat([data_df, synth_df])
clf = DecisionTreeClassifier(
max_depth=7, random_state=0) #3
clf.fit(train_df.drop(columns=['Real']), train_df['Real'])
pred = clf.predict(train_df.drop(columns=['Real']))
confusion_matrix(train_df['Real'], pred) #4
The listing provides a compact Python implementation using pandas, NumPy, scikit-learn's breast-cancer dataset, a decision-tree classifier, and confusion-matrix tooling. section 3.2.5
