A simulation creates synthetic data one record at a time. Each record represents an action or state at a point in time, and later records depend on the world established by earlier ones. This makes simulation useful when independently generated rows would miss the order and context that make data plausible. section 11.4
Consider a business that buys and sells goods while maintaining inventory. A purchase of an item that has not been selling, or a sale of an item not in stock, may be unusual because of what happened before it—not because the transaction looks unusual on its own. Separate purchasing and sales agents can act on the current state, while an outlier-producing process introduces rare, different behavior. section 11.4
To construct and use this kind of test data:
- Model the state and ordinary agents, then generate transactions in time order so each action can affect what happens next. section 11.4
- Add a distinct process for behavior you want to detect, such as a purchaser whose transactions form an escalating pattern even if individual purchases are not drastically larger. section 11.4 section 12.1
- Generate multiple plausible datasets and test whether a detector reliably finds the known behavior; random variation helps reveal what can and cannot be detected. section 12.1
Listing 11.6 Simulation representing one company for 1 year
class Purchacer:
def consider_purchase(self): #1
global inventory
global transactions
if (current_datetime.hour < 9) or \
(current_datetime.hour > 17): #2
return
if np.random.random() > 0.01: #3
product_id = np.random.choice(range(num_items_in_inventory))
if inventory[product_id] > 1000: #4
return
supplier_id = \
np.random.choice(range(num_suppliers)) #5
unit_cost = np.random.exponential() * 20.0
num_purchased = np.random.randint(10, 200)
total_cost = num_purchased * unit_cost
inventory[product_id] += num_purchased
transactions.append([
'Purchase', self.staff_id,
supplier_id, product_id,
current_datetime.strftime("%Y-%m-%d %H:%M"),
num_purchased, unit_cost, total_cost,
inventory[product_id]])
The one-year company simulation shows how a purchaser checks working hours and inventory before adding a purchase transaction. section 11.4
class RoguePurchaser: #1
def consider_purchase(self):
if (current_datetime.hour < 9) or (current_datetime.hour > 17):
return
if (current_datetime.day == 28) and \
(current_datetime.month not in \
self.extra_purchase_months): #2
product_id = 5
supplier_id = 10
unit_cost = np.random.exponential() * 250.0
trend_factor = np.log2(current_datetime.month+1) + 1
num_purchased = int(np.random.randint(50, 60) * trend_factor)
total_cost = num_purchased * unit_cost
inventory[product_id] += num_purchased
transactions.append([
'Purchase', self.staff_id, supplier_id,
product_id,
current_datetime.strftime("%Y-%m-%d %H:%M"),
num_purchased, unit_cost, total_cost,
inventory[product_id]])
self.extra_purchase_months.append(current_datetime.month)
elif np.random.random() > 0.01:
product_id = np.random.choice(range(num_items_in_inventory))
if product_id == 0: #3
return
if inventory[product_id] > 1000:
return
supplier_id = np.random.choice(range(num_suppliers))
unit_cost = np.random.exponential() * 20.0
num_purchased = np.random.randint(10, 200) #4
total_cost = num_purchased * unit_cost
inventory[product_id] += num_purchased
transactions.append([
'Purchase', self.staff_id, supplier_id, product_id,
current_datetime.strftime("%Y-%m-%d %H:%M"),
num_purchased, unit_cost, total_cost,
inventory[product_id]])
The rogue purchaser illustrates a separate transaction-generating process with recurring purchases, changing quantities and higher sampled unit costs. section 11.4
Simulation is particularly useful for testing collective outliers: a segment can differ in aggregate even when its rows are individually unremarkable. Its advantage over simply duplicating or deleting records is that the intended anomalous behavior is known. That advantage depends on how well the simulated processes resemble the real setting. Simulations are difficult to set up, and synthetic tests should not replace later work with real data or use the same mechanism to define an outlier and validate its detection. section 12.1 section 11.6
