Multiprocessing runs work concurrently across multiple processes. A parent process creates and manages child processes, then distributes work to them. Unlike threads, separate processes can execute Python bytecode concurrently because each process has its own global interpreter lock. section 1.4.2 section 1.5
Use multiprocessing primarily for CPU-intensive work. Marking CPU-bound functions async does not make them run concurrently: the event loop remains blocked unless the work is moved into a process pool. section 1.4.2 section 2.7.1
A practical multiprocessing workflow follows these ideas:
- Create a process with a target function, call start to begin it, and call join to wait for completion. section 1.4.2
- Use ProcessPoolExecutor when work can be divided among independent inputs; submit or map distribute calls and provide results as they complete. section 6.3.1
- Serialize results sent back to the parent process, and protect shared state with locks when multiple processes may access it simultaneously. section 6.6 section 6.7
Running a child process
A minimal program defines hello_from_process, creates multiprocessing.Process(target=hello_from_process), starts the child, prints from the parent, and joins the child. The separate process IDs demonstrate that the functions run in different processes. section 1.4.2
Scaling CPU-bound counting
A ProcessPoolExecutor can apply count to numbers such as 1, 3, 5, 22, and 100000000. Small counts finish quickly, while the large count takes longer, illustrating independent work distributed across pooled processes. section 6.3.1
Listing 6.4 Process pool executors
import time
from concurrent.futures import ProcessPoolExecutor
def count(count_to: int) -> int:
start = time.time()
counter = 0
while counter < count_to:
counter = counter + 1
end = time.time()
print(f'Finished counting to {count_to} in {end - start}')
return counter
if __name__ == "__main__":
with ProcessPoolExecutor() as process_pool:
numbers = [1, 3, 5, 22, 100000000]
for result in process_pool.map(count, numbers):
print(result)
Inspect the complete ProcessPoolExecutor example to see pool creation, mapped inputs, timing output, and the main-process guard. section 6.3.1
