What you will learn in Grokking GPU Programming
With Examples in CUDA
Chapter by chapter
Chapter 1: Why parallel programming?
This introduction explains why automatic hardware speedups gave way to multicore processors and identifies GPUs as a large source of processing units.
What you learn in chapter 1 →Chapter 2: Your first parallel program
This introduction previews the CUDA concepts covered in the chapter and the grading-room analogy that will guide the first program.
What you learn in chapter 2 →Chapter 3: Scaling up: from one block to many
This introduction explains scaling array processing beyond one block by spreading threads across multiple blocks.
What you learn in chapter 3 →Chapter 4: The Map pattern
This introduction defines the Map pattern, illustrates its independent per-element work, and relates it to GPU execution.
What you learn in chapter 4 →Chapter 5: The Reduction pattern
This introduction contrasts reduction with Map and illustrates parallel summation through partial sums and pairing rounds.
What you learn in chapter 5 →Chapter 6: The Scan pattern: Hillis-Steele
Introduces scan as a way to compute running results and contrasts its output and algorithmic tradeoffs with reduction and work-efficient scanning.
What you learn in chapter 6 →Chapter 7: Work-efficient scan: Blelloch
This introduction contrasts Blelloch’s work-efficient scan with Hillis-Steele and recaps the scan concepts and Hillis-Steele mechanism.
What you learn in chapter 7 →Chapter 8: The Histogram pattern
This introduction illustrates lost updates in shared results and contrasts them with earlier patterns that reduce write collisions.
What you learn in chapter 8 →Chapter 9: The Stencil pattern
Introduces stencil computation through a parallel image blur and contrasts its neighbor reads with histogram write conflicts.
What you learn in chapter 9 →Chapter 10: Memory access patterns
This introduction identifies global-memory access patterns as a possible reason a correct, well-organized GPU kernel runs slowly.
What you learn in chapter 10 →Chapter 11: Shared memory
This introduction explains why shared memory can speed up GPU computation and identifies the common collaboration pattern used in earlier kernels.
What you learn in chapter 11 →Chapter 12: Parallel matrix multiplication
This introduction explains the data bottleneck in naive matrix multiplication and how tiling with shared memory can reduce repeated global-memory access.
What you learn in chapter 12 →Chapter 13: Read-only data optimization
This introduction explains why repeated warp-wide reads can strain ordinary caches and how constant memory provides a dedicated cache for read-only data.
What you learn in chapter 13 →Chapter 14: Warps and occupancy
Introduces warp-based thread execution and its performance implications when threads take different paths.
What you learn in chapter 14 →Concepts explained in this book
- shared memoryShared memory is a fast, explicitly managed memory space accessible to threads within a GPU thread block.
- constant memoryConstant memory is a read-only CUDA memory space backed by device DRAM, with a limited address space and a dedicated cache near the streaming multiprocessors.
- scanThe scan pattern computes cumulative results for the prefixes of an ordered sequence.
- occupancyGPU occupancy is the ratio of active warps on a streaming multiprocessor to the maximum number of warps it can support.
- strideMemory stride is the distance between memory locations accessed by successive elements of an access pattern.
- exclusive scanAn exclusive scan computes each prefix result using only elements before the current position, beginning with the operation's identity element.
- arithmetic intensityArithmetic intensity is the number of floating-point operations performed per byte transferred to and from memory.
- mapThe map pattern applies an operation independently to each element of a collection.
- threadA thread is a sequence of instructions that performs a unit of computation.
- convolutionConvolution is an operation that combines input values with corresponding kernel weights over a neighborhood to compute outputs.
- histogram patternThe histogram pattern counts input values into corresponding categories or bins, combining contributions that may come from multiple threads.
- warpA warp is a GPU execution group whose threads typically execute the same instruction concurrently on different data.
- reduction operationThe reduction pattern combines many input values into one result using an associative operation, often by combining partial results in parallel.
- atomic operationAn atomic operation performs a read-modify-write or similar memory update as an indivisible action with respect to other threads.
- data reuseData reuse is the practice of retaining or sharing a value so that multiple computations can use it without repeatedly fetching it from its original storage.
- __syncthreads()__syncthreads is a CUDA barrier operation that makes threads in one block wait until all threads in that block reach the barrier.
- blelloch algorithmThe Blelloch algorithm is a work-efficient parallel algorithm for computing a scan.
- privatizationA technique that gives each parallel worker or group of workers a private copy of data, which can be merged after processing.
- identity elementAn identity element is a value that leaves the result of an operation unchanged when combined with another value.
- constant cacheThe constant cache is a dedicated on-chip GPU cache that stores data read from constant memory for fast access, especially when threads request the same value.
- parallelismParallel computing is the execution of computational work simultaneously across multiple processing units.
- matrix-matrix multiplicationMatrix multiplication is the algebraic operation that combines rows of one matrix with columns of another to produce a matrix of dot products.
- race conditionA race condition is a concurrency error in which unsynchronized reads and writes let one thread overwrite a value before another thread reads it.
- bounds checkingA bounds check is a conditional test that verifies an index lies within a data structure's valid range before access or computation.
- data parallelismData parallelism performs the same operation concurrently on multiple pieces of data.
- warp divergenceBranch divergence is a GPU execution inefficiency that occurs when threads in the same warp take different control-flow paths.
- workWork is the total amount of computation performed by an algorithm, commonly measured by its operation count.
- row-major orderRow-major order is a multidimensional array layout in which elements of each row are contiguous in memory.
- memory trafficMemory traffic is the amount of data transferred between memory and processing units during computation.
- Hillis-Steele scanThe Hillis-Steele algorithm computes a parallel scan through logarithmically many rounds, doing work proportional to the input size times the number of rounds.
- atomicAddatomicAdd is a CUDA atomic operation that adds a value to a memory location without allowing concurrent updates to interfere.
- tileA tile is a smaller rectangular block of elements taken from a larger matrix or other data structure.
- CUDA kernelA CUDA kernel is a function launched on a GPU to execute concurrently across many GPU threads.
- memory coalescingMemory coalescing combines memory requests from GPU threads into fewer transactions when their accesses are suitably grouped.
- global indexA global index identifies a thread's corresponding element across all blocks in a CUDA grid.
- cudaMemcpyToSymbolcudaMemcpyToSymbol is a CUDA runtime API function that copies data to a named device symbol.
- block sizeThreads per block is the configured number of GPU threads assigned to each CUDA thread block in a kernel launch.
- SIMTSIMT is a GPU execution model in which groups of threads execute instructions together while operating on separate data.
- tilingTiling is a technique that partitions data or computation into smaller blocks to enable reuse and reduce memory traffic.
- memory boundA memory-bound kernel is a GPU kernel whose performance is limited primarily by memory data transfer.
- tile sizeTile size is the number of rows and columns in a tile used by a tiled computation.
- GPUA GPU is a processor designed to execute many operations concurrently, especially data-parallel operations.
- shared memory bank conflictA bank conflict occurs when threads in the same GPU warp access different data assigned to the same shared-memory bank, requiring serialized service.
- __constant____constant__ is a CUDA variable qualifier that places a variable in the device's constant memory address space.
- ceiling divisionCeiling division is division rounded upward to the smallest integer greater than or equal to the exact quotient.
- cudaDeviceSynchronizecudaDeviceSynchronize is a CUDA runtime API function that blocks the host until previously issued work on the device has completed.
- tiled matrix multiplicationTiled matrix multiplication divides matrix operands into smaller blocks that can be computed using local data reuse.
- compute-boundCompute-bound describes a computation whose performance is limited primarily by the processor's computational capacity.
- paddingPadding adds extra elements to a data structure, commonly with neutral values, to meet an algorithm's size requirements.
- spanDepth is the number of sequential steps in a parallel algorithm assuming unlimited processors; it is also called span.
- CUDACUDA is NVIDIA's parallel computing platform and programming model for executing general-purpose computations on GPUs.
- CPUA CPU is a general-purpose processor that executes program instructions.
- associative binary operationAn associative operation is an operation whose result is unchanged when its operands are regrouped.
- warp schedulerWarp scheduling is the GPU hardware process of selecting warps for execution on available processing resources.
- GPU memoryDevice memory is memory located on a GPU and accessible to GPU computations.
- naive matrix multiplication kernelNaive matrix multiplication assigns one GPU thread to each output element, with that thread computing the full dot product for that element.
- memory hierarchyA memory hierarchy is an arrangement of storage levels with differing capacities, access latencies, and bandwidths.
- double bufferingDouble buffering uses two buffers alternately to separate data being read from data being written.
- CUDA data lifecycleThe CUDA data lifecycle is the five-stage GPU workflow of allocating device memory, copying data in, computing, copying results out, and freeing device memory.
- threadIdx.xthreadIdx.x is a CUDA-provided per-thread coordinate identifying a thread's position along the x dimension within its block.
- struct of arraysA structure of arrays stores each record field in a separate array, with corresponding positions belonging to the same record.
- data-reuse testThe data-reuse test is a criterion for deciding whether a kernel benefits from shared memory based on whether data is read repeatedly.
- predicationPredication is a technique for avoiding divergent control flow by conditionally executing instructions or computing alternatives and selecting a result.
- warp-uniform branchA warp-uniform branch is a conditional branch for which all threads in a warp take the same path.
- thread limitA thread limit is the maximum number of threads that can be assigned to one CUDA block.
- register spillingRegister spilling is the compiler's placement of register values in slower memory when register demand exceeds the available or permitted registers.
- cooperative loadingCooperative loading is a technique in which multiple threads work together to load data into a shared storage area.
- down-sweepThe down-sweep is the phase of the Blelloch scan that propagates accumulated results through the scan tree to produce prefix results.
- SIMD hardwareA SIMD/AVX unit is processor hardware that applies one vector instruction to multiple data elements at once.
- synchronizationSynchronization is the coordination of concurrent computations so that their execution observes required ordering or completion constraints.
- barrierA barrier is a synchronization mechanism that makes participating threads wait until the required threads reach a specified point.
- invariantAn invariant is a property that remains true at each stage of an algorithm.
- array of structsAn array of structures stores records as an array in which each element contains multiple fields belonging to one record.
- three-step recipeThe three-step recipe is a shared-memory workflow of loading data, collaborating on it within a block, and writing results back.
- multi-block scanA multi-block scan strategy computes a scan over data spanning multiple GPU thread blocks by combining block-level results.
- floating-point operation countA floating-point operation is an arithmetic operation performed on floating-point values.
- speedupSpeedup is the ratio of a computation's sequential execution time to its parallel execution time.
- dynamic shared memoryDynamic shared memory is shared memory whose capacity is specified when a CUDA kernel is launched.
- static shared memoryStatic shared memory is shared memory whose capacity is fixed in a CUDA kernel declaration.
- column-major orderColumn-major order is a multidimensional array layout in which elements of each column are contiguous in memory.
- latencyLatency is the time required to complete one operation or job.
- massively parallel computationMassively parallel computation performs the same or similar independent operations concurrently across a very large amount of data.
- throughputThroughput is the amount of work completed per unit of time or over an overall execution period.
- const __restrict__ qualifiersA const __restrict__ pointer is a pointer declaration that marks its data as read-only and promises the referenced data is not aliased through another pointer.
- thread blockA block is a CUDA launch hierarchy unit whose threads are grouped and scheduled for execution on GPU hardware.
- data dependencyA data dependency occurs when one computation requires a value produced or modified by another computation.
- block dimensionBlock dimensions are the configured extents of a CUDA thread block along its coordinate axes.
- up-sweepThe up-sweep is the bottom-up phase of the Blelloch scan that combines values into subtree totals.
- shared memory broadcastA memory broadcast supplies one value read from a single address to multiple threads.
- reduction treeA reduction tree is a tree structure in which values are combined through successive levels to produce an aggregate.
- write-after-read raceA write-after-read hazard occurs when a write may overwrite a location before a preceding read from it is complete.
- zero paddingZero padding extends an input with zero-valued elements beyond its boundary.
- GPGPUGPU programming is the practice of writing software that executes computational work on graphics processing units.
- ridge pointA ridge point is the arithmetic-intensity threshold that separates memory-bound from compute-bound performance for a given bandwidth and compute rate.
- Amdahl’s LawAmdahl’s Law models the maximum speedup obtainable when the problem size is fixed and a fraction of the work remains sequential.
- cudaMemcpycudaMemcpy is a CUDA runtime API function that copies data between host and device memory or between device memory regions.
- global function qualifier__global__ is a CUDA function qualifier that declares a function callable from the host and executable on the device as a kernel.
- resource contentionContention occurs when multiple threads compete to access or update the same resource, potentially limiting parallel performance.
- alignmentMemory alignment is the placement of data at an address satisfying the boundary requirements of its type or hardware access.
- snapshot ruleThe Snapshot Rule is a memory-access analysis technique that considers the locations accessed by all warp threads at the same moment.
- register fileA register file is a collection of fast on-chip registers that stores values for a processor's active threads.
- histogramA histogram is a collection of counts recording how many values fall into each category or bin.
- __launch_bounds____launch_bounds__ is an NVIDIA-specific CUDA compiler attribute that specifies a kernel's maximum threads per block and optionally a minimum number of blocks per multiprocessor.
- hardware registerA GPU register is a tiny, fast storage location private to an individual thread.
- kernel boundaryA kernel boundary is the transition between GPU kernel executions that, in the described ordinary same-stream launch sequence, provides a grid-wide synchronization point and makes prior writes visible.
- producer-consumer dependencyA dependency in which a consumer's read of shared data must follow the producer's write to that data.
- read-after-write hazardA read-after-write hazard occurs when a read may happen before a preceding write to the same location is complete.
- Gustafson's lawGustafson’s Law models scalable speedup by allowing the problem size to grow as more processors are added.
- associativityAssociativity is the property that regrouping operands does not change the result of an operation.
- CUDA execution hierarchyThe CUDA execution hierarchy consisting of grids, blocks, and threads.
- two-dimensional thread layoutA 2D thread layout arranges the threads of a CUDA block along two dimensions, giving each thread two coordinates.
- vector additionVector addition combines two vectors by adding their corresponding components.
- select instructionA select instruction chooses one of two values based on a condition without splitting warp control flow.
- floating-point operations per secondFLOPS measures the rate of floating-point operations performed per second.
- sequential processingSequential execution performs computational steps one at a time rather than concurrently.
- memory transactionA memory transaction is a unit of memory service that processes one or more requests between a processor and memory.
- memory bankA memory bank is an independently accessed division of memory that can serve requests concurrently with other banks.
- matrixA matrix is a two-dimensional arrangement of elements organized by rows and columns.
- CUDA kernel launch syntaxCUDA launch syntax specifies the execution configuration used to launch a kernel, including its blocks and threads.
- gridA grid is a CUDA launch hierarchy containing the thread blocks of a kernel execution.
- swap-and-addSwap-and-add is a scan operation that passes a parent value to a child and adds the child’s prior value to the parent.
- SPMDSPMD is a programming model in which one program is executed by many parallel threads, with each thread operating on different data.
- scattered memory access patternA scattered memory access pattern occurs when threads access memory locations that are not close together, often requiring separate memory transactions.
- thread communicationThread communication is the exchange of data between threads during concurrent execution.
- embarrassingly parallelAn embarrassingly parallel computation consists of independent work items that can be parallelized with little or no coordination.
- L2 cacheAn L2 cache is a hardware-managed cache that stores copies of data to reduce the cost of memory access.
- streaming multiprocessorA streaming multiprocessor is a GPU hardware unit that schedules and executes groups of GPU threads using its execution resources.
- one thread per elementOne thread per element is a data-mapping approach that assigns one parallel thread to each element of an input collection.
- effective memory bandwidthEffective bandwidth is the data-transfer rate a computation achieves after accounting for the efficiency of its memory-access pattern.
- read-modify-write operationA read-modify-write is an operation sequence that reads a value, computes an update, and writes the result back.
- thread layoutA thread layout is the arrangement of threads in a CUDA block along one, two, or three axes.
- nvccnvcc is NVIDIA's CUDA compiler driver for compiling CUDA source files into executable programs.
- thread execution orderThread execution order is the sequence in which concurrently running threads reach and execute operations.
- integer divisionInteger division is division whose result is an integer obtained by truncating the fractional part of the quotient.
- critical pathThe critical path is the longest chain of dependent steps that must execute sequentially in a computation.
- kernel launchA kernel launch starts execution of a GPU kernel across a specified number of threads.
- out-of-bounds array accessAn out-of-bounds access is an attempt to read or write outside the valid range of a data structure or memory allocation.
- cudaGetLastErrorcudaGetLastError is a CUDA runtime API function that returns the latest CUDA error status for the calling host thread.
- memory bandwidthMemory bandwidth is the rate at which data can be transferred between a processor and memory.
- work efficiencyWork efficiency describes an algorithm's computational work relative to that required by an efficient computation of the same problem.
- cudaMalloccudaMalloc is a CUDA runtime API function that allocates memory on a GPU device.
- power wallThe power wall is a physical limit on increasing processor speed caused by power consumption and heat dissipation.
- asynchronous programmingAsynchronous execution allows an operation to proceed without requiring the caller to wait for its completion before continuing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
