Multiprocessing
Multiprocessing is a programming technique that executes work in multiple operating-system processes, enabling concurrent or parallel execution across available processor resources.
Python Concurrency with asyncio
Multiprocessing is a concurrency technique in which multiple processes perform work concurrently, typically with a parent process managing child processes.
- How it is used
- Uses multiprocessing to parallelize CPU-bound work and extend asyncio when asynchronous tasks would block the event loop.
- How deep it goes
- Explains parent-child processes, start and join, process pools, serialization, shared state, and event-loop-per-process designs.
- Examples
- Demonstrates child-process greetings, Fibonacci computation, pooled counting, and database queries handled by one event loop per process.
- What you can do after
- Builds process-based workers, distribute CPU-bound calls through ProcessPoolExecutor, and combine process pools with asyncio.
Read how Python Concurrency with asyncio explains multiprocessing →
Fast Python: High performance techniques for large datasets
- How it is used
- Uses multiprocessing to run a pure-Python, in-memory MapReduce implementation across available CPU cores.
- How deep it goes
- Works through an executor substitution, diagnoses a pickling failure, and gives communication guidance without developing shared-memory or low-level locking techniques.
- Examples
- Uses a ProcessPoolExecutor MapReduce worker whose lambda fails to pickle and whose named-function replacement succeeds.
- What you can do after
- Apply it via ProcessPoolExecutor to parallelize coarse-grained Python work and diagnose worker-transfer or communication bottlenecks.
Read how Fast Python: High performance techniques for large datasets explains multiprocessing →
Learn Concurrent Programming with Go
An operating-system capability in which multiple tasks are managed so that more than one can make progress on available processor resources.
- How it is used
- Keeps processor resources productive by switching to other tasks when the current task waits for input or I/O.
- How deep it goes
- Explains job, ready, running, and I/O waiting states, then demonstrates process coordination through a buffered pipe.
- Examples
- Uses a large-file string search and a curl-to-wc UNIX pipeline to illustrate scheduling, process IDs, buffering, and termination.
- What you can do after
- Evaluate/understand how operating systems schedule tasks and coordinate command-line processes through pipes.
Read how Learn Concurrent Programming with Go explains multiprocessing →
Related concepts
- global interpreter lockcontrasts with · The global interpreter lock is a mutual-exclusion mechanism in CPython that permits only one thread at a time to execute Python bytecode within a process.
- parallelismkind of · Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.
- concurrent.futurescontrasts with · concurrent.futures is a Python standard-library module providing high-level interfaces for asynchronously executing and coordinating callable tasks with threads or processes.
- operating systemused by · An operating system is system software that manages computer hardware and provides services and interfaces for application programs.
- process poolhas part · A process pool is a managed collection of worker processes that executes submitted tasks concurrently.
- compute-bounduses · A computation is compute-bound when its performance is limited primarily by available processor or arithmetic capacity rather than memory transfer, input/output, or other resources.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.