A memory profiler measures how much memory program actions consume, helping locate expensive lines, functions, objects, and peak usage. Python offers both high-level summaries and line-by-line profiling. section 7.1 section 7.1.2
Profile a Python function with memory-profiler:
- Install the package with pip install memory-profiler. section 7.1.2
- Place @profile above the function whose memory consumption you want to inspect. section 7.1.2
- Run the script with python -m memory_profiler your_script.py. section 7.1.2
- Read Mem usage, Increment, Occurrences, and peak memory to identify costly operations. section 7.1.2
Profiling a large model-training workflow
A decorated function reads a 40-million-row ads dataset, engineers one-hot features, and trains logistic regression. The initial workflow peaks near 3,384 MB, with reading the dataset adding about 1,774 MB and feature creation about 779 MB. A chunked workflow profiled with the same tool peaks at 267 MB and completes in about 70 seconds, compared with more than two hours. section 7.1.2 section 7.5.5
Listing 7.4 This code is a simple example of reading in a sample of the ads data, and then return a filtered version of the dataset (where the field, click = 0). We use the @profile decorator above the function so that memory-profiler will profile this function when calling the library from the terminal.
import pandas as pd #1
from sklearn.linear_model import LogisticRegression
import time
@profile #2
def train_logit_model(filename: str): #3
ad_frame = pd.read_csv(filename) #4
print("Finished reading dataset...\n\n") #4
input_features = ad_frame[[ #5
"site_category",
"banner_pos"]]
sc_features = pd.\
get_dummies(input_features.\
site_category)
final_features = pd.concat([ #6
sc_features,
ad_frame.banner_pos])
logit_model = LogisticRegression( #7
random_state = 0)
print("Start training...\n\n") #8
logit_model.fit(ad_frame[final_features],
ad_frame.click)
return logit_model #9
start = time.time()
logit_model = train_logit_model( #10
"data/ads_train_data.csv")
end = time.time()
print("Finsihed running in ",
end - start, " seconds") #11
Inspect a focused Python example using @profile on an ads-model training function. section 7.1.2

