Variance measures how widely values in a data set spread around their mean. Calculate it by finding each value’s deviation from the mean, squaring those deviations, and averaging the results. Squaring prevents positive and negative deviations from canceling each other and gives larger deviations more influence. For a population, the calculation uses every available value; n is the data-set size, xi an individual value, and μ the mean. section 6.1.1
Standard deviation is the square root of variance. Both describe variability, but standard deviation is used directly to assess whether repeated network connections occur at consistent intervals: low values indicate a tighter, more regular pattern. Variance can be calculated alongside it as an additional spread measure. section 6.1.1 section 6.1.5
To use variance while hunting for regular beaconing behavior:
- Collect timestamped network-connection events with source IP, destination IP, and destination port. section 6.1.1
- Group events that share the same source IP, destination IP, and destination port, then calculate time differences between consecutive connections. section 6.1.1 section 6.1.5
- Calculate variance, standard deviation, and connection count for each group; retain one row per group after removing duplicates. section 6.1.5
- Prioritize sufficiently repeated groups with low standard deviation, then investigate the destination, traffic details, and responsible process. section 6.1.5
Listing 6.14 Calculating standard deviation, variance, and count
df['std1'] = df.groupby(['src_ip', 'dest_ip', \
'dest_port'])['time_diff_sec'].transform('std') #1
df['var1'] = df.groupby(['src_ip', 'dest_ip', \
'dest_port'])['time_diff_sec'].transform('var') #2
df['count1'] = df.groupby(['src_ip', 'dest_ip', \
'dest_port'])['time_diff_sec'].transform('count') #3
This DataFrame pattern computes variance, standard deviation, and count from connection intervals for each source-IP, destination-IP, and destination-port group. section 6.1.5
Checking repeated connections
For connections between 10.0.0.6 and 208.80.154.224, the interval data contains 4,077 connections. Its reported standard deviation is 78.269678 and variance is 6126.142499. A line graph of intervals over time and a histogram of their distribution help inspect the pattern behind those summary values. section 6.2 section 6.3
