Multiprocessing

Q: How deeply does Fast Python: High performance techniques for large datasets cover multiprocessing?

Works through an executor substitution, diagnoses a pickling failure, and gives communication guidance without developing shared-memory or low-level locking techniques.

As explained in Fast Python: High performance techniques for large datasets

Multiprocessing is a programming technique that executes work in multiple operating-system processes, enabling concurrent or parallel execution across available processor resources.

Multiprocessing runs work in separate processes rather than Python threads. The Global Interpreter Lock (GIL) restricts parallel execution of Python code in threads, but it does not restrict separate processes. Multiprocessing can therefore use multiple CPU cores for pure-Python work. section 1.3.1 section 3.6

A process-based MapReduce worker

For an in-memory MapReduce implementation, `concurrent.futures.ProcessPoolExecutor` provides a process-based alternative to `ThreadPoolExecutor`. The import can change to `from concurrent.futures import ProcessPoolExecutor as Executor`, but that change alone may leave the reduce stage apparently frozen. Inspecting completed futures reveals a pickling error when the worker function is a lambda. Defining it with `def counter(emitted): return emitted[0], sum(emitted[1])` resolves that specific failure. section 3.4 section 3.4.1

Processes communicate through pickle, so code and data sent to workers must be transferable by it or handled separately. File pointers, database connections, sockets, and some foreign-library objects need particular care. Serialization can also be slow enough to erase a parallel speedup. Favor coarse-grained jobs with little communication, and check whether communication—not computation—has become the bottleneck. Shared memory, low-level locks, and complex communication patterns make parallel code harder to debug. section 3.4.1 section 3.6

How Fast Python: High performance techniques for large datasets covers it

Cover of Fast Python: High performance techniques for large datasets

liveBook manning.com

Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Python Concurrency with asyncioMultiprocessing is a concurrency technique in which multiple processes perform work concurrently, typically with a parent process managing child processes.
liveBook manning.com

Learn Concurrent Programming with GoAn operating-system capability in which multiple tasks are managed so that more than one can make progress on available processor resources.

Compare how 3 books explain multiprocessing →

Related concepts

global interpreter lockcontrasts with · The global interpreter lock is a mutual-exclusion mechanism in CPython that permits only one thread at a time to execute Python bytecode within a process.
parallelismkind of · Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.
concurrent.futurescontrasts with · concurrent.futures is a Python standard-library module providing high-level interfaces for asynchronously executing and coordinating callable tasks with threads or processes.
operating systemused by · An operating system is system software that manages computer hardware and provides services and interfaces for application programs.
process poolhas part · A process pool is a managed collection of worker processes that executes submitted tasks concurrently.
compute-bounduses · A computation is compute-bound when its performance is limited primarily by available processor or arithmetic capacity rather than memory transfer, input/output, or other resources.

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.