liveBook · Manning

Parallelism

As explained in Software Engineering for Data Scientists

Parallelization is the execution of computational work concurrently across multiple processing resources.

Parallelization speeds up computational work by splitting a problem into sub-problems that can run at the same time. It matters when growing data makes common tasks slow: scraping many files, creating features across large datasets, generating predictions with low latency, or downloading many URLs. The goal is faster execution for work that can be divided safely, not simply running every line of code simultaneously. section 6.0 section 6.2

Figure 6.2 This view shows an example of a parallelization workflow for downloading the same example 1500 files. Here, we can split up the downloads into separate workflows that occur at the same time. The benefit is that the original task of getting all of the files downloaded occurs much faster because multiple file downloads occur simultanaouesly in parallel.
ch6 parallelization workflow

A sequential download waits for each file before starting the next; parallel downloads divide 1,500 files into simultaneous workflows, reducing the total wait. section 6.2

A useful first question is whether parts of the task are independent. Downloading files can be split into separate downloads. A random forest can build multiple trees at once because its trees are independent. Hyperparameter searches can also evaluate candidate settings in parallel. Speed improvements differ by algorithm and workload. section 6.2 section 6.2.3

Two primary forms organize parallel work:

  • Vectorization operates on a collection of values at once instead of processing one element at a time. This fits operations across dataset columns, such as deriving features from existing variables. section 6.2
  • Multiprocessing runs multiple instances of an application or program. In Python, multiple Python instances can work on a task to complete it faster. section 6.2

Train a random forest on multiple cores

For an artists-popularity model using follower count and number of genres, set RandomForestRegressor(n_jobs=1) for single-process training, then raise n_jobs to 3 to use three cores. The measured run drops from about 172.4 seconds to 106.8 seconds. Avoid setting n_jobs higher than the CPUs available. Setting n_jobs to -1 uses one less than the machine’s core count. section 6.2.3

Listing 6.18 This code modifies the previous listing just by increase the number of cores used to 3.
import pandas as pd #1
from sklearn.ensemble import RandomForestRegressor
import time

artists = pd.read_csv("data/artists.csv") #2


artists = artists.\ #3
          dropna().\
          reset_index(drop = True)

artists['num_genres'] = artists.genres.map(len) #4

features = ["followers", "num_genres"] #5

start = time.time() #6
forest_model = RandomForestRegressor(n_estimators = 300,
    min_samples_split = 40,
    max_depth = 3,
    verbose = 1,
    n_jobs = 3
    )

forest_model.fit(artists[features], #7
    artists.popularity) #7
end = time.time()
print(end - start)

This listing isolates the practical change: the model setup is retained while n_jobs changes from 1 to 3, making the number of cores explicit. section 6.2.3

The same idea applies to hyperparameter tuning. A search grid for a random forest can use an n_jobs setting while trying parameter combinations. More cores can substantially reduce runtime: the illustrated search falls from 678.4 seconds with one core to 291.4 seconds with three, though results depend on the task. section 6.2.3

How Software Engineering for Data Scientists covers it

How deep it goes
Defines parallel work, distinguishes vectorization and multiprocessing, and demonstrates n_jobs-based multi-core model training and tuning.
Examples
Contrasts 1,500 sequential and simultaneous downloads, then times random-forest training and hyperparameter searches on the artists dataset.
What it assumes
Uses Python, pandas, and scikit-learn examples while introducing parallelization from a plain-language definition.
What you can do after
Operate/configure parallel model training and hyperparameter searches by choosing an appropriate n_jobs value for available CPUs.
How it connects
Connects scaling with vectorization, multiprocessing, random forests, feature engineering, and hyperparameter tuning.

Also explained in

Compare how 55 books explain parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.