What you will learn in CUDA for LLMs
Building fast transformer kernels
Chapter by chapter
Chapter 1: When PyTorch just isn’t enough
This introduction connects effective GPU use with LLM performance and positions raw CUDA as a direct way to control GPU execution.
What you learn in chapter 1 →Chapter 2: Your first CUDA program
This introduction previews a first complete CUDA program and identifies CPU-side host code as part of the program that launches the vectorAdd kernel.
What you learn in chapter 2 →Chapter 3: Naive kernels
This chapter introduction establishes correctness-first naive CUDA kernels and a CPU-oracle pattern for verifying their outputs.
What you learn in chapter 3 →Chapter 4: Training and verifying neural nets in raw CUDA
This chapter introduces a progression from a PyTorch reference implementation to a verified and optimized CUDA neural network.
What you learn in chapter 4 →Chapter 5: Transformer inference in CUDA
The chapter introduces transformer inference and contrasts its single-token performance profile with training.
What you learn in chapter 5 →Chapter 6: Optimizing our transformer kernels
The introduction motivates GPU-aware optimization and previews a profiling-driven approach to improving transformer kernels.
What you learn in chapter 6 →Chapter 7: Tensor cores
Introduces tensor cores as specialized hardware for accelerating the matrix multiplications central to deep learning workloads.
What you learn in chapter 7 →Chapter 8: Flash Attention
This introduction contrasts tensor-core compute throughput with attention’s data-movement bottleneck and previews Flash Attention.
What you learn in chapter 8 →Chapter 9: Quantization
The chapter introduction explains how quantization can make large models fit on smaller GPUs and trade some accuracy for faster, lower-memory inference.
What you learn in chapter 9 →Chapter 10: Distributed computing
This introduction explains how PCIe and NVLink interconnects affect communication performance within a multi-GPU server.
What you learn in chapter 10 →Chapter 11: CUTLASS
The introduction positions CUTLASS as a way to gain more control over GEMM performance than a black-box library provides.
What you learn in chapter 11 →Chapter 12: Now what?
This introduction explains the book’s deliberately narrow focus on durable CUDA fundamentals and frames the final chapter as a guide to continued learning.
What you learn in chapter 12 →Concepts explained in this book
- CUDA kernelA kernel is a function launched for parallel execution by many GPU threads.
- quantization scaleA quantization scale is the factor relating an integer quantization step to the corresponding interval in the original numerical range.
- matrix-matrix multiplicationGeneral matrix multiplication (GEMM) computes a scaled matrix product and accumulates it into a scaled existing matrix.
- softmaxSoftmax is a function that converts real-valued scores into nonnegative values that sum to one.
- parallelismParallel processing is the concurrent execution of multiple computations across processing resources.
- quantizationQuantization represents numerical values using a limited set of discrete levels, often to reduce model storage and computation costs.
- layer normalizationLayer normalization normalizes a neural network's activations across the features of each input, typically with learned scaling and shifting.
- shared memoryShared memory is a low-latency on-chip GPU memory region used for cooperation among threads.
- warpA warp is a group of GPU threads scheduled to execute instructions together on a streaming multiprocessor.
- tilingTiling is a computation technique that partitions data into smaller blocks to improve locality and reuse.
- memory coalescingMemory coalescing is a GPU memory-access technique in which neighboring threads access neighboring memory locations so requests can be combined.
- naive CUDANaive CUDA is an initial CUDA implementation that lacks important performance optimizations and may use inefficient memory-access patterns.
- threadA CPU thread is an operating-system-managed execution context that runs instructions on a CPU.
- online softmaxOnline softmax is an incremental softmax algorithm that updates normalization statistics while processing elements so the input need not be materialized in full.
- matrix-vector multiplicationGEMV is a general matrix-vector multiplication operation with optional scaling of the product and output vector.
- quantization errorA rounding error is the difference between an exact value and its finite-precision representation or computed approximation.
- allreduceAllReduce is a collective communication operation that reduces values across participants and distributes the complete result to each participant.
- wmma operationA WMMA operation is a warp-level matrix multiply-accumulate operation that uses GPU tensor cores.
- reference implementationA reference implementation is a correctness-oriented implementation that establishes expected behavior rather than optimized performance.
- asymmetric quantizationAsymmetric quantization maps a numerical range to quantization levels without requiring the range to be symmetric around zero.
- memory bandwidthMemory bandwidth is the rate at which data can be transferred between memory and a processor.
- symmetric quantizationSymmetric quantization maps values around zero to signed integer representations using a scale without an offset.
- WGMMAWGMMA is an NVIDIA GPU instruction family that performs asynchronous matrix multiply-accumulate operations across a warp group.
- inferenceModel inference is the execution of a trained machine-learning model on input data to produce predictions or other outputs.
- FlashAttentionFlashAttention-2 is a version of FlashAttention that improves GPU utilization through reduced non-matrix-multiplication work and improved parallelism and work partitioning.
- top-kTop-k selection identifies the k highest values in a collection and optionally their indices.
- data parallelData parallelism distributes different portions of input data across devices that perform corresponding computations on model replicas.
- autoregressive generationAutoregressive generation produces a sequence one token at a time, conditioning each new token on preceding tokens.
- NCCLNCCL is NVIDIA's library of optimized collective communication primitives for multi-GPU and multi-node computing.
- Tensor CoreA tensor core is a specialized GPU execution unit designed to accelerate matrix operations commonly used in machine learning.
- all-gatherAll-gather is a collective communication operation that gathers each participant's data and distributes the combined result to all participants.
- reduce_scatterReduce-scatter is a collective communication operation that reduces data across participants and distributes distinct portions of the result among them.
- CUTLASSCUTLASS 3.x is a major version series of NVIDIA's C++ template library for GPU linear algebra.
- attention headAn attention head is an individual attention computation that projects inputs to queries, keys, and values and produces an output.
- feedforward networkA transformer feed-forward network independently transforms each position's representation using linear layers and a nonlinear activation.
- transformer blockA transformer block is a transformer unit containing attention and feed-forward sublayers together with residual connections and layer normalization.
- arithmetic intensityComputational intensity is the ratio of floating-point operations performed to bytes transferred.
- key-value cacheA KV cache stores previously computed attention keys and values for reuse during sequence generation.
- gridA CUDA thread grid is the complete collection of thread blocks launched for one kernel execution.
- dequantizationDequantization is converting quantized integer representations back into a higher-precision numerical representation.
- decoder-only transformerA decoder-only transformer is a transformer architecture that generates sequences using causal attention over preceding tokens.
- quantization calibrationQuantization calibration determines quantization parameters from representative data to map values into a lower-precision representation.
- mixture of expertsA mixture of experts is a neural-network architecture that routes inputs to selected specialized subnetworks called experts.
- warp-level reductionA warp-level reduction combines values held by threads in a warp into an aggregate such as a sum or maximum.
- per-channel quantizationChannel-wise quantization uses separate quantization parameters for individual channels or output features.
- reduction operationA reduction combines multiple input values into an aggregate through an operation such as a sum.
- static quantizationStatic quantization computes activation quantization parameters before inference using calibration data and reuses them for subsequent inputs.
- quantization granularityQuantization granularity specifies the size or structure of the value groups that share quantization parameters.
- causal maskA causal mask identifies which sequence positions are permitted as attention targets so that future tokens cannot be attended to.
- embedding lookup tableAn embedding table is a learned matrix that maps discrete token identifiers to dense vector representations.
- multi-head attentionMultihead attention runs multiple attention heads and combines their outputs with a projection.
- residual connectionA residual connection adds a sublayer's input to its output, providing a direct path through the network.
- scaled-dot product attentionScaled dot-product attention forms query-key scores, scales and normalizes them, and uses the resulting weights to combine values.
- trainingModel training is the process of adjusting a model's parameters using data and an optimization procedure.
- calibration dataA calibration dataset is a representative collection of samples used to estimate quantization parameters.
- thread blockA CUDA thread block is a schedulable group of threads that can cooperate using shared resources.
- numerical stabilityNumerical stability is the property of a computation that limits errors or invalid results caused by finite-precision arithmetic.
- collective communicationA collective operation is a coordinated communication operation involving a group of distributed processes or devices.
- block quantizationBlock-wise quantization assigns a separate scale factor to each spatial block of a tensor.
- WMMA fragmentA WMMA fragment is a warp-distributed representation of a matrix tile used as an operand or accumulator in warp-level matrix operations.
- broadcast operationA collective broadcast distributes data from a root participant to all participants in a group.
- memory layoutA memory layout is the arrangement of data elements in memory addresses.
- gpu video memoryDevice memory is GPU-resident storage accessible to device-side computation such as CUDA kernels.
- fully sharded data parallelFully sharded data parallel is a distributed training technique that shards model parameters, gradients, and optimizer states across devices to reduce per-device memory use.
- zero-pointA quantization zero point is the integer value that corresponds to real zero in an affine quantization mapping.
- tensor parallelismTensor parallelism distributes parts of a tensor computation across multiple devices and combines their results.
- reduce collectiveA reduce collective combines values from participating processes or devices and delivers the result to a designated root.
- access patternA memory access pattern is the sequence and arrangement of memory locations read or written by a computation.
- tileA tile is a bounded subset of an array or matrix processed as a unit.
- dynamic quantizationDynamic quantization computes at least some quantization parameters at runtime from the values being processed.
- backpropagationBackpropagation computes gradients by propagating derivatives backward through a neural network.
- block-per-row strategyA block-per-row strategy assigns a separate GPU thread block to process each row of data.
- int8 quantizationINT8 is an 8-bit integer numerical format used to represent values in quantized computations.
- profilingProfiling is the measurement of program execution behavior to identify resource usage and performance bottlenecks.
- tensor-wise quantizationTensor-wise quantization applies a shared set of quantization parameters to an entire tensor.
- int4A 4-bit integer is an integer representation occupying four bits, distinct from CUDA's int4 vector type.
- gatherGather collects data from multiple participants and concatenates it at one destination.
- ring algorithmA ring collective algorithm exchanges data among participants arranged in a logical cycle to perform collective communication.
- latencyLatency is the elapsed time between a request and the corresponding response or output.
- expert routingExpert routing selects experts to process inputs in a mixture-of-experts model.
- pipeline parallelismPipeline parallelism partitions a model into sequential stages assigned to different devices so that multiple inputs can be processed concurrently.
- FP32Float32 is a 32-bit floating-point numerical format.
- AWQActivation-aware weight quantization is a method that scales weight channels using activation statistics before quantizing them to reduce output error.
- warp groupA warp group is a collection of GPU warps that cooperate in executing certain collective instructions.
- decode phaseDecode is the inference phase that generates tokens sequentially from a model's existing context.
- logitA logit is an arbitrary real-valued score supplied as input to a normalization or classification function.
- PyTorchPyTorch is an open-source machine learning framework with tensor computation and automatic differentiation capabilities.
- float16FP16 is a 16-bit floating-point numerical format commonly used in machine learning computation.
- data transferData movement is the movement of data between locations or memory spaces in a computing system.
- int4 quantizationINT4 quantization represents model values with 4-bit integers to reduce memory consumption at the cost of additional quantization error.
- expert networkAn expert is a subnetwork in a mixture-of-experts model whose learned weights specialize in processing particular inputs.
- network topologyNetwork topology is the arrangement of devices and communication links in a network.
- quantization step sizeQuantization step size is the spacing between adjacent quantization levels on the original numerical scale.
- block-level reductionA block-level reduction combines values from threads within a GPU thread block into an aggregate result.
- percentile-based calibrationPercentile calibration sets quantization ranges using distribution percentiles, potentially clipping extreme values.
- kernel launch configurationA kernel launch configuration specifies the grid and block dimensions for a CUDA kernel execution.
- keyAn attention key is a vector compared with a query to determine the relevance of an associated value.
- queryA query is an attention representation compared with keys to determine attention scores.
- valueAn attention value is a vector combined with other values according to attention weights.
- memory boundA memory-bound operation is an operation whose performance is limited primarily by memory access rather than computation speed.
- FP4FP4 is a four-bit floating-point numerical format used for low-precision representation and computation.
- data reuseData reuse is the repeated use of a loaded value across multiple computations to reduce memory traffic.
- PTXPTX is NVIDIA's virtual instruction set architecture and intermediate assembly language for GPU programming.
- Tensor Memory AcceleratorThe Tensor Memory Accelerator is NVIDIA GPU hardware for asynchronous multidimensional data transfers between global and shared memory.
- kernel launchA kernel launch is the operation that starts execution of a CUDA kernel on GPU threads.
- epilogueAn epilogue is the sequence of operations applied to a kernel's accumulated result before it is written to memory.
- occupancyOccupancy is the ratio of active warps on a GPU streaming multiprocessor to the maximum supported number of active warps.
- vectorizationVectorization is the organization of operations to process multiple data elements together using vector instructions or accesses.
- self-attentionSelf-attention is an attention mechanism that computes relationships among positions in the same input sequence.
- kernel fusionOperator fusion combines multiple computational operations into a single execution unit to reduce overhead or intermediate data movement.
- mean squared errorMean squared error is the average of the squared differences between corresponding predicted or reconstructed and reference values.
- top-k kernelThe top-k kernel is the example CUDA implementation that extracts expert scores and indices for each token.
- mma_syncmma_sync is a CUDA WMMA function that cooperatively performs a warp-level matrix multiply-accumulate operation on fragments.
- vanishing gradient problemThe vanishing-gradient problem occurs when gradients become too small to effectively train earlier layers of a deep network.
- __shfl_down_sync__shfl_down_sync is a CUDA warp shuffle intrinsic that exchanges values between participating threads at a specified downward offset.
- GPUA GPU is a massively parallel processor designed to perform graphics and general-purpose computations.
- tile shapeA thread block tile shape specifies the dimensions of a computation tile assigned to a GPU thread block.
- integer quantizationInteger quantization represents numerical values with integer levels mapped to an original numerical scale.
- 1D block tiling1D block tiling assigns each thread multiple output elements along one dimension of an output tile, enabling reuse of loaded data.
- correction factorA correction factor rescales the previous exponential sum when the running maximum increases.
- hardware registerA register is a tiny per-thread storage location providing very low-latency access to values during GPU computation.
- fused Flash Attention kernelThe fused Flash Attention kernel is an example CUDA implementation that combines attention computations in one kernel.
- CUDACUDA is NVIDIA's parallel computing platform and programming model for executing general-purpose computations on GPUs.
- transpositionMatrix transposition exchanges the row and column indices of a matrix's elements.
- min-max calibrationMin-max calibration sets the quantization range using the observed minimum and maximum values.
- running maximumA running maximum is the largest value encountered so far in a sequence of processed values.
- producer-consumerA producer-consumer pipeline divides work between producers that prepare data and consumers that process it, coordinating their stages.
- gradientA gradient is a collection of derivatives describing how a function changes with respect to its parameters.
- bottleneckA performance bottleneck is a resource or component whose limited capacity constrains a system's execution speed or throughput.
- GEMV kernelThe GEMV kernel is the example CUDA implementation of matrix-vector multiplication for inference.
- insertion sortInsertion sort orders items by inserting each item into its proper position in a previously ordered subset.
- mixed precisionMixed-precision training uses multiple numerical precisions during model training to reduce resource requirements while maintaining useful numerical behavior.
- clippingClipping, also called clamping, restricts values to a specified interval by replacing out-of-range values with the nearest boundary.
- normalization sumA normalization sum is the sum of unnormalized values used as the denominator when scaling them into a probability distribution.
- boundary checkA boundary check is a conditional test that prevents a computation from accessing data outside its valid range.
- hostA host is the CPU-based system that controls execution and provides resources to an accelerator in a host-device computing model.
- custom CUDACustom CUDA is programming GPU computations specifically for an operation or workload rather than relying solely on framework-provided implementations.
- throughputThroughput is the amount of work a system completes per unit of time.
- NVLinkNVLink 4 is the fourth generation of NVIDIA's NVLink high-speed interconnect technology.
- tcgen05TCGen05 is a Blackwell tensor-core instruction family supporting asynchronous matrix operations and low-precision formats.
- strideA memory access stride is the address distance between successive accesses in an access pattern.
- attentionAn attention mechanism computes weighted combinations of representations according to their relevance to a query.
- shared memory descriptorA shared-memory descriptor encodes the location and layout of an operand in GPU shared memory for an instruction.
- cuBLAScuBLAS is NVIDIA's GPU-accelerated library for basic linear algebra operations.
- nested loopA nested loop is a loop construct placed inside one or more other loop constructs to iterate over multiple dimensions or levels.
- forward passA forward pass computes a neural network's outputs from its inputs and parameters.
- global indexA global index is a unique one-dimensional identifier for a thread's position across all blocks in a kernel launch.
- probability distributionA probability distribution assigns nonnegative probabilities to possible outcomes such that their total is one.
- compute-boundA compute-bound kernel is a GPU kernel whose performance is limited primarily by computational throughput.
- training loopA training loop repeatedly processes training data, computes model outputs and loss, and updates model parameters.
- warp shuffleA warp shuffle is a GPU operation that exchanges register values directly among threads in the same warp.
- sparse activation patternSparse activation executes only a subset of a model's components for each input.
- warp-per-row strategyA warp-per-row strategy assigns each row of data to a cooperating GPU warp.
- thread-per-rowA thread-per-row strategy assigns one GPU thread to process an entire row of data.
- shared-memory reductionA shared-memory reduction combines GPU thread values through successive aggregation steps using shared memory and synchronization.
- gpu coreA GPU core is a simple GPU execution component composed of arithmetic and logic units for parallel numerical operations.
- uncoalesced memory accessAn uncoalesced memory access is a GPU access pattern in which threads' requests cannot be efficiently combined into memory transactions.
- PCIePCIe is a high-speed serial interconnect standard for connecting computer components and peripheral devices.
- CPUA CPU is a processor designed with relatively few powerful cores for sequential tasks and complex control logic.
- memory hierarchyA memory hierarchy is an arrangement of storage levels with different capacities, access latencies, and sharing scopes.
- inline assemblyInline assembly embeds assembly instructions within code written in a higher-level programming language.
- CUTLASS profilerThe CUTLASS profiler is a tool for running and measuring the performance of CUTLASS GPU operations under different configurations.
- quantization rangeA quantization range is the interval of integer values available for representing quantized data.
- varianceVariance measures the average squared deviation of values from their mean.
- matrix multiply-accumulateMatrix multiply-accumulate multiplies matrices and adds the resulting product to an accumulator.
- attention matrixAn attention probability matrix stores normalized attention weights for pairs of sequence positions.
- inner productAn inner product combines two vectors into a scalar, including by summing corresponding elementwise products in the standard case.
- strong scalingStrong scaling evaluates how performance changes when more processing resources work on a fixed-size problem.
- weak scalingWeak scaling evaluates how efficiently a computation runs as its problem size increases with its processing resources.
- anomalyAn outlier is a data value that differs substantially from typical values in a distribution.
- benchmarkA benchmark is a reference workload or measurement basis used to compare performance or correctness.
- CollectiveBuilderCollectiveBuilder is a CUTLASS API for constructing collective kernel components from high-level specifications and architecture-specific choices.
- __syncthreads()__syncthreads is a CUDA block-level barrier that synchronizes participating threads and makes prior memory accesses visible within the block.
- convolutionA convolution is an operation that combines a local sliding filter with input data to produce transformed features.
- max poolingMax pooling selects the maximum value within each local input region, commonly to reduce spatial resolution.
- tree algorithmA tree collective algorithm performs collective communication by passing data along the edges of a logical tree.
- load_matrix_syncload_matrix_sync is a CUDA WMMA operation that synchronously loads a matrix tile from memory into a warp-distributed fragment.
- SXMAn SXM GPU is a GPU packaged for NVIDIA's SXM module form factor.
- exponentA floating-point exponent is the field that determines the order of magnitude of a represented number.
- mantissaA significand is the precision-bearing component of a floating-point number, determining its fine detail within an exponent's range.
- CUDA eventA CUDA event is a GPU stream marker that can be recorded and queried for synchronization or elapsed-time measurement.
- ReLUReLU is an activation function that returns the maximum of zero and its input.
- CuTePython CuTe is a Python-based interface for expressing and generating GPU kernels using CuTe's tensor and layout abstractions.
- mean absolute errorMean absolute error is the average absolute difference between corresponding predicted or reconstructed and reference values.
- chain ruleThe chain rule computes the derivative of a composition of functions by multiplying the derivatives of its constituent functions.
- linear layerA linear layer is a neural-network layer that computes an affine transformation of its inputs using learned weights and a bias.
- address spaceA memory space is a distinct addressable memory domain associated with a computing processor or execution environment.
- hierarchical reductionA hierarchical reduction combines values in stages across progressively larger groups of parallel threads.
- SIMTSIMT is a processor execution model in which multiple threads execute the same instruction stream while individual threads may branch independently.
- cudaDeviceSynchronizecudaDeviceSynchronize is a CUDA runtime API function that waits for previously requested device work to complete.
- vectorized loadingA vectorized memory transfer moves multiple adjacent data elements with a single wide memory instruction.
- alignmentMemory alignment specifies the address boundary or element multiple to which data is arranged for memory access.
- Blackwell architectureBlackwell is an NVIDIA GPU architecture designed for accelerated computing and artificial intelligence workloads.
- affine transformAn affine transformation combines a linear transformation with an additive offset.
- GPGPUGPU programming is the practice of writing software whose computation executes on graphics processing units.
- meanThe mean is the sum of a collection of values divided by their count.
- standardizationStandardization transforms values by subtracting their mean and dividing by their standard deviation.
- shared memory bank conflictA shared memory bank conflict occurs when concurrent GPU memory requests contend for the same shared-memory bank and require additional servicing.
- high-bandwidth memoryHigh-bandwidth memory is stacked DRAM that provides high data-transfer rates through a wide memory interface.
- CUDA programming modelThe CUDA programming model describes how programs divide work between host code and parallel GPU execution.
- ceiling divisionCeiling division computes the smallest integer greater than or equal to the quotient of two integers.
- multi-kernel approachA multikernel parallel approach divides a computation among multiple parallel kernel launches.
- attention weightAn attention weight is a normalized coefficient determining how much an attention value contributes to an output.
- ring bufferA circular buffer is a fixed-size buffer whose storage is reused by wrapping access around to the beginning.
- prefillPrefill is the inference phase that processes an input prompt and prepares its attention cache for generation.
- mmaMMA is a family of PTX matrix multiply-accumulate instructions that use explicitly managed registers.
- cross entropy lossCross-entropy loss measures the discrepancy between a target probability distribution and a predicted probability distribution.
- Nsight ComputeNVIDIA Nsight Compute is a profiler for analyzing and optimizing CUDA kernel performance.
- column-major orderA column-major layout stores multidimensional array elements so that the leftmost dimension varies fastest in linear memory.
- floating-point operations per secondTFLOPS measures computational throughput in trillions of floating-point operations per second.
- Model ServingModel serving is the operation of a deployed model to process inference requests and return results.
- fused multiply-addA fused multiply-add computes a multiplication and addition as one operation with a single final rounding.
- NumPyNumPy is a Python library for numerical computing with multidimensional arrays.
- asynchronous programmingAsynchronous execution allows an operation to proceed independently while subsequent work is issued before it completes.
- accumulatorAn accumulator is a storage location that holds an incrementally updated result during a computation.
- SlurmSLURM is an open-source workload manager and job scheduler for computing clusters.
- MPIMPI is a standardized interface for communication among processes in parallel and distributed programs.
- register-tiling GEMMRegister tiling assigns multiple output elements to each GPU thread and keeps intermediate values in registers for reuse.
- pipelineA pipeline is an execution organization that overlaps successive stages of data movement and computation.
- precisionNumerical precision is the level of detail with which a numerical representation can distinguish values.
- hierarchical tile decompositionHierarchical tiling partitions a computation into nested tiles assigned to different levels of a parallel hardware hierarchy.
- transformer architectureA transformer is a neural-network architecture that processes sequences using attention and position-aware transformations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
