liveBook · Manning

Memory profiler

As explained in Software Engineering for Data Scientists

memory-profiler is a Python library that measures the memory consumption of program execution, including memory usage associated with individual lines or functions.

A memory profiler measures how much memory program actions consume, helping locate expensive lines, functions, objects, and peak usage. Python offers both high-level summaries and line-by-line profiling. section 7.1 section 7.1.2

Profile a Python function with memory-profiler:

  • Install the package with pip install memory-profiler. section 7.1.2
  • Place @profile above the function whose memory consumption you want to inspect. section 7.1.2
  • Run the script with python -m memory_profiler your_script.py. section 7.1.2
  • Read Mem usage, Increment, Occurrences, and peak memory to identify costly operations. section 7.1.2

Profiling a large model-training workflow

A decorated function reads a 40-million-row ads dataset, engineers one-hot features, and trains logistic regression. The initial workflow peaks near 3,384 MB, with reading the dataset adding about 1,774 MB and feature creation about 779 MB. A chunked workflow profiled with the same tool peaks at 267 MB and completes in about 70 seconds, compared with more than two hours. section 7.1.2 section 7.5.5

Listing 7.4 This code is a simple example of reading in a sample of the ads data, and then return a filtered version of the dataset (where the field, click = 0). We use the @profile decorator above the function so that memory-profiler will profile this function when calling the library from the terminal.
import pandas as pd #1
from sklearn.linear_model import LogisticRegression
import time

@profile #2
def train_logit_model(filename: str): #3

    ad_frame = pd.read_csv(filename) #4
    print("Finished reading dataset...\n\n") #4

    input_features = ad_frame[[ #5
        "site_category",
        "banner_pos"]]

    sc_features = pd.\
        get_dummies(input_features.\
        site_category)

    final_features = pd.concat([ #6
        sc_features,
        ad_frame.banner_pos])

    logit_model = LogisticRegression( #7
        random_state = 0)

    print("Start training...\n\n") #8
    logit_model.fit(ad_frame[final_features],
        ad_frame.click)

    return logit_model #9


start = time.time()
logit_model = train_logit_model( #10
    "data/ads_train_data.csv")
end = time.time()
print("Finsihed running in ",
    end - start, " seconds") #11

Inspect a focused Python example using @profile on an ads-model training function. section 7.1.2

How Software Engineering for Data Scientists covers it

How deep it goes
Explains installation, decorators, terminal execution, output columns, peak usage, and practical optimization comparisons.
Examples
Uses ads-data model training, chunked reads, and filtering non-click records to expose line-level memory costs.
What you can do after
Diagnose/debug with memory-profiler by locating high-increment lines and comparing peak memory before and after changes.
How it connects
Connects memory profiling with computational profiling, line-profiler, pandas dataframes, feature engineering, and chunked processing.

Also explained in

Compare how 2 books explain memory profiler →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.