Parallelism
Parallelism is the concurrent execution of multiple computational operations, often across multiple processing resources.
Fast Python: High performance techniques for large datasets
- How it is used
- Uses simultaneous execution to exploit CPU cores, GPU units, or distributed workers for data processing.
- How deep it goes
- Works through Python execution limits and several implementation routes, but defers general parallel-programming techniques.
- Examples
- Tests the idea through MapReduce word counting, a Cython cellular automaton, array doubling, and Dask partition reductions.
- What you can do after
- Apply it via multiprocessing, Cython, or Dask, and diagnose lost parallelism through communication costs and task graphs.
Read how Fast Python: High performance techniques for large datasets explains parallelism →
Grokking Concurrency
Parallelism is the simultaneous execution or simultaneous progress of multiple computational operations.
- How it is used
- Uses simultaneous execution of independent tasks to reduce runtime or increase throughput.
- How deep it goes
- Develops hardware and independence requirements, a chunked Python implementation, and speedup limits; defers fuller treatment of decomposition.
- Examples
- Contrasts laundry machines and cooks, then compares sequential and process-pool password searches.
- What you can do after
- Apply it via Python’s multiprocessing Pool to split an independent search into chunks and compare its runtime with serial execution.
Latency
Parallelism is the simultaneous execution of tasks across multiple processors or other execution units.
- How it is used
- Reduces processing latency by running independent work simultaneously across multiple processors.
- How deep it goes
- Defines true parallel execution, estimates limits with Amdahl’s law, and examines task placement and measured overhead without building a CPU-parallel implementation.
- Examples
- Uses an order-processing dependency chain, independent product-page requests, parallel fanout, and a sequential-versus-Tokio timing comparison.
- What you can do after
- Evaluate/understand it by identifying independent work, estimating speedup limits, and checking measured latency against coordination and tail-latency costs.
Functional Programming in Scala, Second Edition
Parallelism is the execution of multiple parts of a computation at the same time, typically on multiple processor cores or CPUs.
- How it is used
- Models parallel and asynchronous computations as composable Par values instead of directly coordinating threads.
- How deep it goes
- Derives Par, map2 strictness, laws, and generalized combinators, while deferring execution behind a separate interpreter.
- Examples
- Uses divide-and-conquer integer summation, conditional computation choice, and balanced string concatenation folds.
- What you can do after
- Build it from scratch as a composable Par library with combinators, laws, and a separate execution interpreter.
Read how Functional Programming in Scala, Second Edition explains parallelism →
Starting Data Analytics with Generative AI and Python
Parallelized code is software structured so that multiple portions of a computation can execute concurrently.
- How it is used
- Positions parallelism as a possible way to scale analytics computations, subject to workload, hardware, budget, and measured performance.
- How deep it goes
- Works through a threaded bootstrap revision and its slowdown, while deferring cluster synchronization and fault-tolerance internals to established frameworks.
- Examples
- Contrasts optimized and threaded bootstrap timings, then uses independent Euclidean-distance calculations as a better parallelization candidate.
- What you can do after
- Apply it via Python’s ThreadPoolExecutor, then benchmark against the optimized baseline and reconsider workloads where threading adds overhead.
Read how Starting Data Analytics with Generative AI and Python explains parallelism →
Advanced Algorithms and Data Structures
Parallel computing is the execution of multiple computational operations concurrently, typically using multiple processing units.
- How it is used
- Tests whether sequential clustering work can be split across workers to reduce running time.
- How deep it goes
- Works through k-means steps and ideal re-centering operation counts, then identifies result aggregation as a bottleneck.
- Examples
- Uses per-cluster k-means re-centering and coin counting to show independent work and the cost of combining results.
- What you can do after
- Evaluate/understand it by identifying independent algorithm steps and checking whether aggregation will limit the speedup.
Read how Advanced Algorithms and Data Structures explains parallelism →
100 Go Mistakes and How to Avoid Them
Parallel execution is the simultaneous execution of multiple instances of a task, typically on separate processing cores.
- How it is used
- Distinguishes simultaneous task execution from concurrent workflow structure and guides decisions about goroutines and parallel tests.
- How deep it goes
- Works through merge-sort implementations and benchmark results, then gives test scheduling controls; stops short of prescribing a universal workload threshold.
- Examples
- Uses duplicated coffee-shop equipment, sequential and parallel merge sort, benchmark timings, and a paused-and-resumed test log.
- What you can do after
- Evaluate/understand it through sequential-versus-parallel benchmarks, and operate/configure it in tests with t.Parallel() and go test -parallel.
Read how 100 Go Mistakes and How to Avoid Them explains parallelism →
Python Concurrency with asyncio
- How it is used
- Frames parallelization as a performance technique for distinguishing simultaneous execution from task switching in concurrent applications.
- How deep it goes
- Explains cores, the GIL, process pools, chunked MapReduce, reduction, shared state, and locks without detailing process internals.
- Examples
- Uses a two-baker cake analogy, multithreaded HTTP status reads, and parallel word-frequency MapReduce.
- What you can do after
- Build it from scratch by partitioning CPU-bound data, running chunks in process pools, gathering results, and merging partial outputs.
Read how Python Concurrency with asyncio explains parallelism →
Learn Concurrent Programming with Go
Parallelism is the simultaneous execution of multiple computations, typically on multiple processing resources.
- How it is used
- Distinguishes actual simultaneous execution from concurrent task planning and time-shared interleaving.
- How deep it goes
- Explains hardware requirements, scheduler interleaving, Go runtime handling, and Gustafson’s scaling perspective without runtime internals.
- Examples
- Uses request handling, file search, matrix rows, video rendering, I/O waiting, and multicore game workloads.
- What you can do after
- Evaluate/understand it by identifying whether tasks run simultaneously or are merely interleaved on available processing units.
Read how Learn Concurrent Programming with Go explains parallelism →
Kafka for Architects
Parallel processing is the simultaneous execution of multiple data-processing tasks to increase processing capacity or reduce elapsed time.
- How it is used
- Uses topic partitions and consumer groups to divide event-processing work across concurrent consumers.
- How deep it goes
- Explains partition-assignment limits, ordering, lag, worker-pool commits, and Kafka Streams tasks without a low-level implementation walkthrough.
- Examples
- Illustrates partition assignment with an address-change service and distributed stream processing with an orders application.
- What you can do after
- Operate/configure Kafka consumers and Kafka Streams instances with partition limits, ordering, and processing backlog in mind.
Deep Learning with JAX
The execution of computation concurrently across multiple devices or cores.
- How it is used
- Deploys concurrent computation for large-scale training, simulations, and workloads containing independent operations.
- How deep it goes
- Explains pmap() parameters, SPMD execution, collectives, data parallelism, and multihost behavior, while deferring tensor-sharding details.
- Examples
- Demonstrates distributed dot-product calculations with sharded arrays and visualized tensor placement.
- What you can do after
- Apply pmap() to split arrays across devices, configure mapped axes and collectives, and implement data-parallel training.
Apache Kafka in Action
Parallel processing is the simultaneous execution of work across multiple processing units or workers.
- How it is used
- Uses partitioned messages to increase processing throughput across servers and Kafka Streams instances.
- How deep it goes
- Explains partition-to-task assignment, instance distribution, threads, and failover; leaves Apache Flink’s different job management without detailed treatment.
- Examples
- Uses city-based trip data, product-price messages, a two-partition console demonstration, and a 12-partition filtering workload.
- What you can do after
- Operate/configure it by choosing independent partitioned work and distributing Kafka Streams tasks across instances and threads.
Graph Neural Networks in Action
Parallel computing is a technique that divides work among multiple processing units so computations can execute concurrently.
- How it is used
- Uses parallelism to reduce GNN training time by spreading work across processors or machines.
- How deep it goes
- Compares two PyTorch parallelization options and their overhead, while stopping short of a configuration walkthrough.
- Examples
- Compares CPU, GPU, TPU, NPU, and IPU workloads and cites NVIDIA GPUs used on Colab.
- What you can do after
- Evaluate/understand when DistributedDataParallel may suit multi-GPU GNN training better than DataParallel.
Read how Graph Neural Networks in Action explains parallelism →
Rust Servers, Services, and Apps
Parallelism is the simultaneous execution of multiple computations, typically on separate processors or cores.
- How it is used
- Uses parallel execution to improve processing speed and handle simultaneous requests efficiently across available processors.
- How deep it goes
- Defines simultaneous task execution and contrasts concurrency, illustrating CPU assignment without detailing parallelism implementation.
- Examples
- Illustrates two Rust functions assigned to separate CPUs and discusses processing multiple incoming requests.
- What you can do after
- Understand how simultaneous tasks can use multiple CPUs or cores to improve execution and request-processing efficiency.
Read how Rust Servers, Services, and Apps explains parallelism →
Data Pipelines with Apache Airflow, Second Edition
Execution of independent tasks concurrently rather than one after another.
- How it is used
- Runs independent data-pipeline branches together while respecting task dependencies and operational limits.
- How deep it goes
- Explains graph independence, asynchronous waiting, and specific concurrency controls, but does not provide a complete tuning procedure.
- Examples
- Uses independent weather and sales branches, two busy DAGs sharing a global limit, and external jobs monitored asynchronously.
- What you can do after
- Operate/configure it by identifying limiting Airflow settings and reserving worker capacity for tasks that need it.
Read how Data Pipelines with Apache Airflow, Second Edition explains parallelism →
Financial AI in Practice
Parallel processing is the simultaneous execution of multiple computational tasks across processing resources.
- How it is used
- Speeds fraud-graph preparation and community analysis, and reduces runtime in larger financial-data pipelines.
- How deep it goes
- Shows threaded task submission and result collection, then sketches pipeline-level partitioning; it does not benchmark speedups or build a production-scale platform.
- Examples
- Uses transaction-edge preparation, per-community density and node counts, partitioned investment research, and concurrent clickstream and ledger extraction.
- What you can do after
- Apply it via ThreadPoolExecutor to prepare graph edges or analyze communities, while keeping aggregation and dependent pipeline joins explicit.
Learn Go with Pocket-Sized Projects
Parallel programming is a programming technique in which multiple computations execute concurrently to perform work at the same time.
- How it is used
- Uses parallelism to overlap independent work, especially branching exploration and background tasks.
- How deep it goes
- Explains goroutines, scheduling, communication, and tradeoffs, but defers detailed differences between goroutines and system threads.
- Examples
- Illustrates parallelism with cooking two pots and exploring alternative maze paths.
- What you can do after
- Apply it via Go goroutines and channels while judging independence, communication overhead, and uncertain performance gains.
Read how Learn Go with Pocket-Sized Projects explains parallelism →
Code Like a Pro in Rust
Parallelization is the execution of independent computational work concurrently across multiple threads or processing units.
- How it is used
- Guides the choice between threaded computation, concurrent async tasks, and sequential execution for independent Rust workloads.
- How deep it goes
- Explains Tokio scheduling choices and benchmarks Rayon iterator conversions; leaves OS-level thread execution details aside.
- Examples
- Compares blocking and nonblocking sleeps under Tokio, then benchmarks Rayon against sequential summing-squares and regular-expression searches.
- What you can do after
- Apply it via Tokio task spawning or Rayon parallel iterators, and benchmark whether threaded execution improves the workload.
C# Concurrency: Asynchronous and multithreaded programming
Parallel execution is the simultaneous execution of multiple operations by separate processing resources.
- How it is used
- Uses parallel execution to finish independent processing sooner, especially when handling many client requests or item-level operations.
- How deep it goes
- Compares Thread, thread-pool, asynchronous, and Parallel implementations with benchmarks, but defers detailed queue design and delivery semantics.
- Examples
- Demonstrates mail merge, thread-per-message execution, sleeping-thread benchmarks, and asynchronous delay benchmarks.
- What you can do after
- Apply it via C# threads, thread-pool tasks, asynchronous APIs, or Parallel, then measure overhead and limit concurrency appropriately.
Read how C# Concurrency: Asynchronous and multithreaded programming explains parallelism →
Building LLM Applications with DSPy
The overlapping execution of multiple operations so they can progress at the same time.
- How it is used
- Overlaps independent LM requests in DSPy evaluation and module execution to reduce time spent waiting for responses.
- How deep it goes
- Shows thread configuration and async module calls, including `acall()`, `aforward()`, and task grouping; stops short of detailed `asyncio` instruction.
- Examples
- Uses concurrent capital-question predictions, timed custom-module calls, threaded classifier evaluation, and independently answered subtasks.
- What you can do after
- Apply it via DSPy `Evaluate` threads or module `acall()` methods while accounting for rate limits and debugging costs.
Read how Building LLM Applications with DSPy explains parallelism →
Bayesian Optimization in Action
Parallelism is the execution of independent processes simultaneously to reduce total completion time.
- How it is used
- Uses simultaneous evaluations to increase throughput when black-box objective queries are expensive.
- How deep it goes
- Defines independent concurrent execution and explains batch-query clustering, but stops before detailing batch BayesOpt policies.
- Examples
- Uses neural-network training, drug-recipe experiments, three-process timing, and clustered versus spread-out BayesOpt queries.
- What you can do after
- Apply it via batch BayesOpt by evaluating multiple selected query points simultaneously while avoiding redundant clustering.
Read how Bayesian Optimization in Action explains parallelism →
Learn Haskell by Example
Parallelism is the simultaneous execution of independent computations to reduce elapsed computation time.
- How it is used
- Improves multicore image processing by evaluating independent pixel transformations simultaneously.
- How deep it goes
- Explains independence, measurement, sparks, strategies, and chunking, but does not detail a complete parallel implementation.
- Examples
- Demonstrates grayscale, pixelation, and box-blur mappings applied across PNM images before PNG output.
- What you can do after
- Apply parallel evaluation strategies to uniform pure mappings and diagnose ineffective spark granularity.
Grokking Simplicity
Execution in which multiple actions may occur simultaneously or without a fixed order relative to one another.
- How it is used
- Uses timeline diagrams to reason about ordering, timing, and correctness when asynchronous work runs across multiple timelines.
- How deep it goes
- Explains atomic decomposition, timeline construction, timing comparisons, and coordination, while leaving language-specific constraints to the applicable execution semantics.
- Examples
- Uses cost_ajax() and shipping_ajax() requests, response-time comparisons, and diagrams showing sequential and parallel timelines.
- What you can do after
- Diagnose ordering and timing bugs by decomposing actions, drawing timelines, and coordinating parallel work.
Rust in Action: Systems programming concepts and techniques
Parallelism is the simultaneous execution of computational activities on multiple physical processing cores or other execution units.
- How it is used
- Distinguishes simultaneous execution across CPU cores from concurrency that can occur on one core.
- How deep it goes
- Defines parallelism and contrasts it with blocked-thread scheduling, without developing an implementation of concurrency libraries.
- Examples
- Compares threads taking turns during memory or I/O waits on one core with threads executing on multiple CPUs.
- What you can do after
- Evaluate/understand whether concurrent tasks also execute in parallel on multiple CPU cores.
Read how Rust in Action: Systems programming concepts and techniques explains parallelism →
Refactoring to Rust
Parallelism is the execution of multiple computations simultaneously to perform work concurrently.
- How it is used
- Uses parallelism to divide work across processor cores and improve application throughput.
- How deep it goes
- Defines parallelism against concurrency, illustrates throughput and ordering, and stops short of detailed synchronization mechanics.
- Examples
- Uses four threads on four cores, a two-hammer can crusher, and Python Mandelbrot tasks submitted through ThreadPoolExecutor.
- What you can do after
- Apply parallelism via ThreadPoolExecutor and asyncio.gather to distribute independent computations across execution threads.
Learn Docker in a Month of Lunches, Second Edition
Parallelism is the concurrent execution of multiple computational operations or data transfers.
- How it is used
- Optimizes Docker image pull times by distributing large data across concurrently downloaded layers.
- How deep it goes
- Explains concurrent layer downloads, configurable concurrency, Dockerfile alternatives, and transfer trade-offs.
- Examples
- Compares ML dataset images that copy days 70–79 into one layer or separate layers.
- What you can do after
- Evaluate Docker image transfer performance by building alternative targets and comparing sizes and layer histories.
Read how Learn Docker in a Month of Lunches, Second Edition explains parallelism →
Go in Action, Second Edition
Parallelism is the simultaneous execution of multiple computational activities, typically on separate processors or cores.
- How it is used
- Applies simultaneous execution to independent work, blocking I/O, CPU-bound computation, responsive applications, and real-time processing.
- How deep it goes
- Defines parallelism against concurrency and demonstrates goroutine-based execution, but defers detailed synchronization and completion coordination.
- Examples
- Uses an RSS reader that fetches New York Times and BBC feeds through separate goroutines.
- What you can do after
- Apply it via Go goroutines, channels, and worker pools to overlap independent work and use multiple CPU cores.
Read how Go in Action, Second Edition explains parallelism →
AI Agents in Action, Second Edition
The concurrent execution of independent workloads or subtasks to reduce elapsed processing time.
- How it is used
- Speeds independent workloads in multi-agent systems and repetitive task loops.
- How deep it goes
- Explains fan-out execution, asyncio.gather, batching, timing comparisons, rate limits, and independence checks.
- Examples
- Uses company research, pull-request review, document batches, and five text-transformation tasks as examples.
- What you can do after
- Apply it via asyncio.gather to compare sequential and concurrent task processing while monitoring failures and rate limits.
Read how AI Agents in Action, Second Edition explains parallelism →
The Complete Obsolete Guide to Generative AI
Parallel computation is the simultaneous execution of multiple computational operations.
- How it is used
- Accelerates generative AI and other computationally intensive workloads by executing compatible operations concurrently.
- How deep it goes
- Defines parallel computation and connects it to AI acceleration, but stops short of explaining scheduling or implementation details.
- Examples
- Uses AI matrix calculations, neural-network operations, specialized accelerators, and qubits in superposition as examples.
- What you can do after
- Evaluate how parallel processing and specialized hardware can improve AI training, inference speed, energy use, and deployment feasibility.
Read how The Complete Obsolete Guide to Generative AI explains parallelism →
Deep Learning with PyTorch, Second Edition
The practice of dividing work into tasks that execute simultaneously.
- How it is used
- Accelerates machine-learning training by dividing data or computation across simultaneous tasks, processes, devices, or threads.
- How deep it goes
- Explains task decomposition, thread-process differences, collective communication, and engineering challenges without detailing every parallelism strategy.
- Examples
- Uses a ten-call timing comparison, a jigsaw analogy, PyTorch broadcast and all-reduce, and TorchTitan configurations.
- What you can do after
- Apply it via concurrent.futures and PyTorch distributed operations to compare execution modes and coordinate tensor data across processes.
Read how Deep Learning with PyTorch, Second Edition explains parallelism →
Effective Data Analysis
Parallel processing is the simultaneous execution of separate portions of a computational task.
- How it is used
- Accelerates analytics, statistical modeling, and machine learning on datasets too large or slow for local processing.
- How deep it goes
- Explains task division, cluster execution, result recombination, and fault tolerance, but stops short of implementation details.
- Examples
- Uses Spark SQL, PySpark, and a task-partitioning figure as concrete examples of parallel data processing.
- What you can do after
- Apply it via distributed computing frameworks while accounting for framework syntax, resource limits, and processing costs.
Learn Generative AI with PyTorch
A training approach that computes multiple sequence elements simultaneously rather than strictly sequentially.
- How it is used
- Accelerates Transformer training so models can learn from large datasets and support large language models.
- How deep it goes
- Explains simultaneous versus sequential processing and positional encodings, but stops short of device-level implementation details.
- Examples
- Uses Transformer language translation and contrasts it with sequential RNN processing.
- What you can do after
- Evaluate why Transformers train faster than RNNs and why positional encodings accompany parallel input processing.
Read how Learn Generative AI with PyTorch explains parallelism →
Spring AI in Action
Parallelization is a technique that executes multiple processing paths concurrently and aggregates their results.
- How it is used
- Uses parallel paths to answer related questions concurrently within an agent workflow before producing an aggregate response.
- How deep it goes
- Implements worker execution, thread-pool sizing, result aggregation, summarization, and Spring workflow wiring.
- Examples
- Demonstrates Azul analysis by running player-count and game-mechanics actions before summarizing their outputs.
- What you can do after
- Apply it via Java CompletableFuture, executor services, injected actions, and Spring bean configuration.
C++ Concurrency in Action, Second Edition
Parallelism is the simultaneous execution of multiple tasks, typically to use available hardware and improve performance.
- How it is used
- Uses simultaneous tasks to exploit available hardware and improve bulk data-processing performance.
- How deep it goes
- Defines the performance-oriented distinction from concurrency and points to library-managed parallelization without detailing its mechanics.
- Examples
- Uses networked processes and standard-library algorithm calls to illustrate ways to increase or delegate parallel work.
- What you can do after
- Evaluate parallelism as a performance goal and identify standard-library algorithms as a route to delegating task division.
Read how C++ Concurrency in Action, Second Edition explains parallelism →
Introduction to Generative AI, Second Edition
Parallelization is the execution of independent subtasks or tool calls concurrently rather than sequentially.
- How it is used
- Uses concurrent actions to reduce the slowness of sequential tool execution in complex agent tasks.
- How deep it goes
- Explains the principle and task dependencies through a workflow example, but does not detail implementation mechanics.
- Examples
- Uses a pickleball news website whose frontend work and API research proceed in parallel.
- What you can do after
- Apply it by separating independent subtasks from actions whose inputs depend on earlier results.
Read how Introduction to Generative AI, Second Edition explains parallelism →
Outlier Detection in Python
Parallelism is the execution of multiple computational operations concurrently rather than one after another.
- How it is used
- Reduces outlier-detection training or prediction time by executing independent work concurrently.
- How deep it goes
- Explains process controls, detector independence, batch splitting, and sequential trade-offs without detailing parallel-runtime internals.
- Examples
- Uses KDD Cup data, concurrent batch prediction, and sequential Isolation Forest, LOF, and GMM filtering.
- What you can do after
- Apply it via n_jobs, multiprocessing, SUOD, or concurrent batch execution while comparing sequential workload reduction.
AI Agents and Applications
Parallel processing is the simultaneous execution of the same or related operations on multiple independent inputs.
- How it is used
- Accelerates document summarization and web research by executing independent chain operations concurrently.
- How deep it goes
- Explains chunk and search parallelization through map() and composition, while stopping short of low-level scheduling internals.
- Examples
- Uses document chunks, parallel web searches, an Astorga research query, and LCEL chain diagrams and listings.
- What you can do after
- Apply it via LangChain and LCEL to parallelize independent summarization or research steps and aggregate their outputs.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Parallelization is the execution of independent computational operations concurrently across multiple processing resources.
- How it is used
- Accelerates XGBoost training by distributing each boosting model’s feature-column computations across multiple CPU cores.
- How deep it goes
- Explains column blocks and serial boosting, but stops short of detailing scheduling or split-finding algorithms.
- Examples
- Uses XGBoost CPU utilization and column-block training as concrete examples of parallel execution.
- What you can do after
- Operate XGBoost through its Scikit-learn API while benefiting from internal multi-core training.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains parallelism →
Related concepts
- concurrencycontrasts with · Concurrency is the structuring or execution of multiple computational activities so their progress overlaps in time, whether or not they run simultaneously.
- sequential processingcontrasts with · The execution of operations one after another in a defined order, without overlapping their progress.
- diminishing returnsproduces · Diminishing returns is a pattern in which successive increases in an input produce progressively smaller incremental gains in an outcome.
- threaduses · A thread is an independently schedulable sequence or context of execution within a program or process.
- data parallelincludes · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.