A dataset is a collection of related records prepared for analysis or model training. Its usefulness depends on whether the records, fields, labels, and scope fit the question being investigated. Network-connection events can form datasets for detecting regular beaconing intervals; labeled DNS queries can form training data for identifying tunneling traffic. section 6.1.1 section 9.2.1
Build a dataset around the behavior to measure. For beaconing analysis, group connection events that share source IP address, destination IP address, and destination port, then store the time between subsequent connections. Calculate variance or standard deviation for each group and investigate datasets with low standard deviation, which indicate consistent timing. section 6.1.1
When preparing event data for unsupervised analysis, use a repeatable workflow:
- Load raw events from a local CSV file or a production data store such as Splunk, Humio, or Elasticsearch. section 8.1.2
- Keep fields needed for the investigation, including application, source and destination addresses and ports, timestamps, byte counts, and source type. section 8.1.2
- Filter for connection groups with sufficiently high counts, normalize timestamps and numeric types, and sort records by time. section 8.1.2
- Calculate derived fields such as interconnection time, standard deviation, variance, counts, and inbound-versus-outbound byte differences. section 8.1.2
- Treat fields, columns, and features as interchangeable names for values extracted from or calculated from raw records. section 8.1.2
From connection records to model features
A DataFrame containing 320,044 connection-event rows and 11 retained columns can be narrowed to high-count source/destination/port groups. From those records, time_diff_sec, std1, var1, count1, and bytes_diff become candidate features. Not every original column needs to become a useful model feature. section 8.1.2
Training datasets require an additional check: labels and features must support the prediction task. Relevant, recent data is preferred; training features should match the features later submitted for prediction; and labeled classes should be as balanced as possible. Perfect data is uncommon, so collection choices must reflect the threat variants that matter. section 9.2.1
DNS tunneling training data
A labeled DNS-query dataset can classify regular domain names separately from traffic produced by dns2tcp, dnscapy, Iodine, and tuns. To build a reliable training set, account for variant-specific payload encoding, response size, query type, duplicate events, and throttling time; a single tunneling label may also combine multiple tunneling-tool classes. section 9.2.1
Listing 8.3 Calculating new variables
count_threshold = 100
df = df.groupby(['src_ip', 'dest_ip', 'dest_port']).filter \
(lambda x : len(x)>count_threshold)
df = df.reset_index()
print(len(df.index), "records with count >", count_threshold)
df['timestamp'] = pd.to_datetime(df['timestamp'], format='mixed')
df['endtime'] = pd.to_datetime(df['endtime'], format='mixed')
df['epoch_timestamp'] = df['timestamp'].astype('int64') // 10**9
df['epoch_endtime'] = df['endtime'].astype('int64') // 10**9
df = df.sort_values(by=['epoch_timestamp'], ascending=True)
df['epoch_timestamp'] = df['epoch_timestamp'].astype(int)
df['epoch_endtime'] = df['epoch_endtime'].astype(int)
df['bytes'] = df['bytes'].astype(int)
df['bytes_in'] = df['bytes_in'].astype(int)
df.dtypes
df['time_diff_sec'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['epoch_timestamp'].transform(lambda x: x - x.shift(1))
df['bytes_diff'] = df['bytes_in'] - df['bytes_out'] #1
df['std1'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['time_diff_sec'].transform('std')
df['var1'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['time_diff_sec'].transform('var')
df['count1'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['time_diff_sec'].transform('count')
df[['var1','std1']].sort_values(by=['std1'], ascending=True)
This preparation listing shows how to filter connection groups, convert event values, order timestamps, and derive timing and byte-difference features in a DataFrame. section 8.1.2
