liveBook · Manning

Multiprocessing

As explained in Software Engineering for Data Scientists

Multiprocessing is the execution of computational tasks in separate operating-system processes, typically enabling use of multiple CPU cores.

Multiprocessing runs multiple instances of an application at the same time to finish a task faster. In Python, that means launching separate Python processes and dividing work among them. It is a form of parallelization: instead of waiting for one operation to finish before starting the next, independent operations can proceed simultaneously. This can help when a data science workflow must perform many expensive operations, such as collecting data, processing a large file collection, or generating results under low-latency expectations. section 6.0 section 6.2 section 6.7

A process is an instance of a program, whereas a thread is a smaller unit within a process that shares that process’s resources. Multiprocessing therefore differs from multithreading, asynchronous programming, and vectorization. Choose it when the same function can be applied independently to many inputs and separate Python processes can handle those inputs in parallel. section 6.2 section 6.2.2

A practical multiprocessing workflow is:

  • Check available CPUs with multiprocessing.cpu_count(). The number of concurrent processes is limited by available CPUs. section 6.2.2
  • Put the repeated work in a function that accepts one input, such as a filename. section 6.2.2
  • Create a multiprocessing.Pool with a chosen process count. When four cores are assumed, a pool of three leaves one core free to avoid consuming too many resources. section 6.2.2
  • Use pool.map to apply the function across the input list, then close the pool to shut down the Python processes. section 6.2.2

OCR receipt images in parallel

Optical character recognition (OCR) provides a focused case for multiprocessing. A sequential workflow reads each receipt image with cv2 and calls pytesseract.image_to_string one file at a time; the illustrated run takes about 185 seconds. The parallel version keeps the same scrape_text function but creates Pool(3) and maps that function over the image files. Separate Python processes handle OCR work, and their results are combined. The example runs in roughly half the time, showing how parallelizing independent files can shorten a batch job. section 6.2.2

Figure 6.4 This diagram visualizes how Python’s multiprocessing package creates separate Python processes to tackle the OCR task, followed by combining the results from each individual process. The result of doing this for the OCR task is much faster than using just a single process.
ch6 multiprocessing

This process diagram makes the division of OCR work visible: separate Python processes handle individual tasks before their results are combined. section 6.2.2

Faster execution is not automatic: multiprocessing is one option within a broader efficiency toolkit. Vectorization operates on collections at once, caching trades memory for saved computation, and profiling helps find slow code locations. Multiprocessing is specifically useful when work can be split into concurrent Python-process tasks, such as data collection or image-text extraction. section 6.2 section 6.7

How Software Engineering for Data Scientists covers it

How deep it goes
Demonstrates CPU checking, Pool creation, map execution, and shutdown, without detailing operating-system process internals.
Examples
Uses pytesseract OCR across receipt-image files, comparing sequential processing with a three-process Pool.
What it assumes
Assumes basic Python functions, imports, file lists, and package installation while introducing multiprocessing calls directly.
What you can do after
Apply it via Python’s multiprocessing library to map an independent function across files with a bounded process pool.
How it connects
Contrasts multiprocessing with parallelization, vectorization, multithreading, asynchronous programming, caching, and profiling.

Also explained in

Compare how 4 books explain multiprocessing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Software Engineering for Data Scientists. The section links point to the parts of the book it draws on.