A pandas DataFrame is a two-dimensional data structure: a table of rows and labeled columns. It can be read like a spreadsheet, but its rows can represent individual events and its columns the fields extracted from those events. This makes a DataFrame a practical working surface for turning raw connection records into data that can be filtered, transformed, summarized, and investigated. section 6.1.5 section 8.1.2
For network-event analysis, load only the fields needed for the question: timestamps, source and destination addresses, ports, application, event type, and byte counts. Records can arrive through a data-store API or from local CSV or JSON files; after loading, the same DataFrame-oriented workflow applies. A DataFrame may hold hundreds of thousands of event rows, so selecting relevant columns early keeps the investigation focused. section 6.1.5 section 8.1.2
A repeatable DataFrame workflow for investigating repeated connections is:
- Load raw events into a DataFrame, then retain the columns needed for the investigation. section 6.1.5 section 8.1.2
- Inspect column data types and convert timestamps and numeric values before calculating with them. section 6.1.5 section 8.1.2
- Group rows by source IP, destination IP, and destination port to retain connections occurring above a count threshold. section 6.1.5 section 8.1.2
- Create derived columns such as interconnection time, standard deviation, variance, count, and byte difference. section 6.1.5 section 8.1.2
- Sort, filter, and deduplicate the result to identify destination addresses or connection patterns worth deeper review. section 6.1.5
From connection events to candidate beaconing activity
Start with connection events in which each row is an event. Group the DataFrame by src_ip, dest_ip, and dest_port, and calculate the time difference between consecutive timestamps within each group. Add standard deviation, variance, and count as columns, then keep high-count groups with low timing variation. Dropping duplicate source-destination-port combinations produces a compact candidate list for checking destination context, associated web activity, certificates, or processes. section 6.1.5
Columns are also features when preparing machine-learning input. Imported fields such as addresses, ports, timestamps, and byte counts can be retained, while calculated columns such as time_diff_sec, std1, var1, count1, and bytes_diff capture properties of a connection. Not every available column becomes useful, so feature extraction follows the initial DataFrame preparation. section 8.1.2
Listing 6.6 Collecting events from the Humio data store
import humioapi #1
import pandas as pd #2
api = humioapi.HumioAPI(**humioapi.humio_loadenv()) #3
stream = api.streaming_search( #4
query="sourcetype=stream:tcp OR \ #5
sourcetype=stream:udp \
| findTimestamp(field=timestamp, \
as=epoch_timestamp) \ #6
| findTimestamp(field=endtime, \
as=epoch_endtime) \ #7
| select([epoch_timestamp, epoch_endtime, \
host.name, sourcetype, app, src_ip, \
src_port, dest_port, dest_ip, bytes, \
bytes_in, bytes_out])", #8
repo='Threat_Hunting', #9
start="-24h@h", #10
stop="now" #11
)
df = pd.DataFrame(stream) #12
print(len(df.index), „records fetched.") #13
Inspect the API-based collection pattern that streams matching network events into a pandas DataFrame after selecting the fields needed for analysis. section 6.1.5
Listing 8.3 Calculating new variables
count_threshold = 100
df = df.groupby(['src_ip', 'dest_ip', 'dest_port']).filter \
(lambda x : len(x)>count_threshold)
df = df.reset_index()
print(len(df.index), "records with count >", count_threshold)
df['timestamp'] = pd.to_datetime(df['timestamp'], format='mixed')
df['endtime'] = pd.to_datetime(df['endtime'], format='mixed')
df['epoch_timestamp'] = df['timestamp'].astype('int64') // 10**9
df['epoch_endtime'] = df['endtime'].astype('int64') // 10**9
df = df.sort_values(by=['epoch_timestamp'], ascending=True)
df['epoch_timestamp'] = df['epoch_timestamp'].astype(int)
df['epoch_endtime'] = df['epoch_endtime'].astype(int)
df['bytes'] = df['bytes'].astype(int)
df['bytes_in'] = df['bytes_in'].astype(int)
df.dtypes
df['time_diff_sec'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['epoch_timestamp'].transform(lambda x: x - x.shift(1))
df['bytes_diff'] = df['bytes_in'] - df['bytes_out'] #1
df['std1'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['time_diff_sec'].transform('std')
df['var1'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['time_diff_sec'].transform('var')
df['count1'] = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
['time_diff_sec'].transform('count')
df[['var1','std1']].sort_values(by=['std1'], ascending=True)
Inspect a preparation sequence that filters connection groups, converts types, orders events, and derives timing-based fields in a DataFrame. section 8.1.2
