pandas is an open-source Python library for data manipulation and analysis. Its central structure here is the DataFrame: a two-dimensional, spreadsheet-like table whose rows are events and whose columns are extracted fields. section 6.1.5
For beaconing investigations, pandas turns returned network events into a workable dataset. Load records from a data-store query or CSV, then inspect types, convert timestamps and numeric fields, group related connections, and calculate timing statistics. section 6.1.5
Useful DataFrame operations include:
- Load CSV data with read_csv, retaining an original DataFrame for later investigation. section 6.1.5
- Group by source IP, destination IP, and destination port to filter frequent connections and calculate time differences, standard deviation, variance, and counts. section 6.1.5
- Filter, sort, remove duplicates, select columns, and plot timing distributions to narrow candidate destinations. section 6.1.5
Finding repeated connections
A DataFrame of 667,827 network events is grouped by connection endpoints and port. Low timing variation and repeated connections identify destinations for deeper review; later queries isolate traffic to specific IP addresses and web paths. section 6.1.5
Listing 6.8 Filtering events based on a count threshold
count_threshold = 100 #1
df = df.groupby(['src_ip', 'dest_ip', 'dest_port'])\
.filter(lambda x: len(x) > count_threshold) #2
df = df.reset_index() #3
print(len(df.index), "records with count >", \
count_threshold) #4
This listing demonstrates grouping connection tuples and retaining only groups above a count threshold. section 6.1.5
