Chapter 13: Read-only data optimization
This introduction explains why repeated warp-wide reads can strain ordinary caches and how constant memory provides a dedicated cache for read-only data.
After this chapter you will be able to
- Repeated Global Memory Reads. Explain why repeated reads of the same global-memory value can still be costly even though the GPU combines identical requests from a warp.
- Dedicated Constant Memory Cache. Describe how constant memory helps with a read-only value that all threads in a warp use.
- Shared Global Memory Reads. Explain what happens when all threads in a warp read the same global-memory address and why the read can still be costly.
- Constant Array Cache Waste. Trace the color-correction example to explain why reading three shared floats from global memory can waste cache space.
- Kernel Arguments and Constant Arrays. Distinguish a kernel pointer argument from the data it points to, and explain how an explicitly constant array differs from passing data as kernel arguments.
- Constant memory basic rules. Describe constant memory’s capacity and who writes and reads it, distinguishing it from shared memory.
- Constant memory read patterns. Explain how reading the same location differs from reading different locations in constant memory.
- Constant Memory Placement. Describe where constant memory and its cache sit in the GPU hierarchy and how threads access constant data.
- Address Space and Cache Capacity. Distinguish the constant-memory address space from the per-SM constant cache and explain why their sizes matter.
- Dedicated Cache Isolation. Explain how the dedicated constant cache differs from general L1/L2 caching for repeated reads.
- Read Only Data Choices. Choose among a scalar kernel argument, a __constant__ array, and a const __restrict__ pointer for the read-only data cases described.
- Constant memory usage flow. Describe the three steps for using constant memory and how a kernel reads the value.
- Const and constant memory. Distinguish host const and a const device pointer from a __constant__ variable.
- Symbol copy arguments. Apply cudaMemcpyToSymbol to copy a scalar or array from the host into a declared constant symbol.
- Constant value lifetime and ordering. Explain how long a copied constant value persists and how an update is ordered in the book's default-stream setup.
- Warp Constant Memory Access. Explain how a warp's constant-memory requests are served when threads read the same address versus different addresses.
- Opportunistic and Guaranteed Broadcasts. Distinguish the global-memory broadcast described in the opening from constant-memory broadcast, including why their guarantees differ.
- Warp constant broadcast. Trace how a warp reads a shared constant and explain what happens when some threads are inactive.
- Broadcasts for multiple components. Explain how broadcast and cache hits work when reading multiple float components.
- Unique address serialization. Explain how a warp's constant-memory reads are served when its threads request different addresses, and apply the rule to repeated addresses.
- Cache locality and serialization. Explain how requests for different addresses in the same cache line affect constant-memory rounds and latency.
- Memory choice for scatter. Justify leaving data in global memory when threads need different addresses, contrasting its handling of requests with constant memory.
- Constant memory scaling pattern. Explain how the three-step constant memory pattern scales an array and why the shared factor suits broadcast.
- Constant memory declaration. Describe where scaleFactor is declared and what the __constant__ qualifier does.
- Constant memory symbol lifetime. Explain why the host passes the constant-memory symbol name directly and why it does not free that memory.
- Constant memory scaling kernel. Trace how active threads scale separate array elements using a shared constant-memory value.
- Scalar argument versus constant. Distinguish passing a uniform scalar as a kernel argument from explicitly declaring it in constant memory.
- Constant memory size limit. Explain what happens when an array exceeds the constant memory region.
- Constant symbol copy mistake. Describe the mistake of passing a constant-memory symbol to cudaMemcpy as a device pointer and its stated consequence.
- Const qualifier distinction. Distinguish plain const from __constant__ in terms of the stated performance consequence.
- Constant memory capacity limit. Explain why exceeding the constant-memory limit causes a build failure and how to avoid it.
- Constant Symbol Copy Errors. Explain why cudaMemcpy fails when used with a __constant__ symbol and how to copy the data correctly.
- Partial Constant Array Updates. Describe how cudaMemcpyToSymbol can update part of a constant array and how its default copy direction can be changed.
- Const versus constant memory. Distinguish a host const and a const pointer parameter from data placed in CUDA constant memory.
- Constant memory declaration and loading. Describe how to replace an ordinary const declaration with data in constant memory and identify the benefit of the fix.
What it builds on
- How GPUs really read memory (chapter 10) · GPU memory fetches and cache lines
- Coalesced access: the ideal case (chapter 10) · Combining memory requests within a warp
- Introducing shared memory (chapter 11) · The role of fast on-chip memory
- Step 1: Allocate device memory (set up desks in the grading room) (chapter 2) · Device allocation with cudaMalloc.
- How the hardware runs your threads: warps and SMs (chapter 2) · The role of SMs in running threads.
- Scattered access: the worst case (chapter 10) · Scattered access patterns
- Step 5: Free memory (clear the desks out of the room) (chapter 2) · Freeing dynamically allocated device memory.
- Moving data: CPU to GPU and back (chapter 2) · The host-to-device and device-to-host data movement workflow.
Section by section
Introduction
This introduction explains why repeated warp-wide reads can strain ordinary caches and how constant memory provides a dedicated cache for read-only data.
- Repeated Global Memory Reads: Explain why repeated reads of the same global-memory value can still be costly even though the GPU combines identical requests from a warp.
Why it matters: Combining requests does not eliminate the cost of repeatedly retrieving data that has been evicted from cache. - Dedicated Constant Memory Cache: Describe how constant memory helps with a read-only value that all threads in a warp use.
Why it matters: A dedicated cache is suited to small read-only values shared across a warp.
Key terms: warp, global memory, cache line, L1/L2 caches, constant memory, read-only, dedicated on-chip cache
13.1 One value, 32 reads
This section explains the cache costs of reading shared constants from global memory and distinguishes kernel arguments from explicitly stored constant data.
- Shared Global Memory Reads: Explain what happens when all threads in a warp read the same global-memory address and why the read can still be costly.
Why it matters: A merged read is not necessarily free of memory-system costs. - Constant Array Cache Waste: Trace the color-correction example to explain why reading three shared floats from global memory can waste cache space.
Why it matters: Small shared inputs can occupy cache space needed by the data each thread processes. - Kernel Arguments and Constant Arrays: Distinguish a kernel pointer argument from the data it points to, and explain how an explicitly constant array differs from passing data as kernel arguments.
Why it matters: Passing an address as an argument does not put the pointed-to values in constant memory.
Key terms: warp, global memory, L1/L2 cache, DRAM, cache line, 32-byte sectors, per-pixel data, kernel argument, pointer, __constant__ array
13.2 How constant memory works
This section introduces constant memory and explains its size, host-write and thread-read roles, and performance under same-location versus different-location reads.
- Constant memory basic rules: Describe constant memory’s capacity and who writes and reads it, distinguishing it from shared memory.
Why it matters: These rules help you recognize what data is suitable for constant memory and how it is used. - Constant memory read patterns: Explain how reading the same location differs from reading different locations in constant memory.
Why it matters: The threads’ access pattern determines whether constant memory can serve them efficiently.
Key terms: constant memory, shared memory, host, serialization
13.3 Constant memory within the hierarchy
This section places constant memory in the GPU memory hierarchy and explains its cache, capacity, performance, and appropriate uses.
- Constant Memory Placement: Describe where constant memory and its cache sit in the GPU hierarchy and how threads access constant data.
Why it matters: This explains how data stored in slow off-chip memory can still be read quickly. - Address Space and Cache Capacity: Distinguish the constant-memory address space from the per-SM constant cache and explain why their sizes matter.
Why it matters: Staying within the address-space limit does not guarantee that all constant data stays in the fast cache. - Dedicated Cache Isolation: Explain how the dedicated constant cache differs from general L1/L2 caching for repeated reads.
Why it matters: The benefit is cache isolation, not an exclusive ability to serve a warp's repeated read. - Memory Hierarchy Relative Speed: Describe the relative speed of registers, constant-cache broadcast hits, shared memory, general caches, and DRAM.
Why it matters: The relative speeds help explain why a cached broadcast can be valuable despite constant data residing in DRAM. - Read Only Data Choices: Choose among a scalar kernel argument, a __constant__ array, and a const __restrict__ pointer for the read-only data cases described.
Why it matters: Matching the data and access pattern to the right declaration avoids unnecessary or unsuitable constant-memory use.
Key terms: constant memory, device DRAM, constant cache, streaming multiprocessors, __constant__, constant memory address space, working set, cache miss, L1, L2
13.4 The three steps for using constant memory
This section explains how to declare, populate, and read CUDA constant memory, including its distinction from const and the ordering of updates.
- Constant memory usage flow: Describe the three steps for using constant memory and how a kernel reads the value.
Why it matters: This sequence lets kernels share a read-only value without passing it as a kernel argument. - Const and constant memory: Distinguish host const and a const device pointer from a __constant__ variable.
Why it matters: Read-only syntax alone does not select the GPU memory region used for constant-cache broadcast. - Symbol copy arguments: Apply cudaMemcpyToSymbol to copy a scalar or array from the host into a declared constant symbol.
Why it matters: Using the right function and argument forms places the host data in the intended constant-memory slot. - Constant value lifetime and ordering: Explain how long a copied constant value persists and how an update is ordered in the book's default-stream setup.
Why it matters: Understanding persistence and ordering prevents a constant update from being mistaken for a per-launch argument or a general synchronization technique. - Constant memory readback: Describe how to check a constant symbol's current value from the host.
Why it matters: Reading a symbol back can help verify that the intended value was copied.
Key terms: __constant__, constant memory, constant cache, broadcast, const, global memory, cudaMemcpyToSymbol, symbol, host-to-device, default stream
13.5 The broadcast mechanism
This section explains constant-memory broadcast within a warp and contrasts it with opportunistic global-memory caching.
- Warp Constant Memory Access: Explain how a warp's constant-memory requests are served when threads read the same address versus different addresses.
Why it matters: The addresses threads choose determine whether constant memory can serve the warp in one step. - Opportunistic and Guaranteed Broadcasts: Distinguish the global-memory broadcast described in the opening from constant-memory broadcast, including why their guarantees differ.
Why it matters: The distinction explains why sharing a value through constant memory differs from relying on ordinary caching.
Key terms: warp, constant memory, broadcast, serialized, opportunistic broadcast, guaranteed broadcast, constant cache, read-only, cache-line fetch
13.6 How broadcast works, step by step
This section traces constant-memory broadcast within a warp and explains its behavior with inactive threads, cache misses, and multiple addresses.
- Warp constant broadcast: Trace how a warp reads a shared constant and explain what happens when some threads are inactive.
Why it matters: This explains how one constant read can serve a whole warp without requiring all its threads to be active. - Constant cache miss behavior: Explain why a broadcasted constant read can still incur a cache fetch.
Why it matters: A single broadcast does not guarantee that the requested value is already cached. - Broadcasts for multiple components: Explain how broadcast and cache hits work when reading multiple float components.
Why it matters: Separate broadcasts do not necessarily mean separate cache misses for neighboring components.
Key terms: warp, constant cache, broadcast, transaction, active thread, miss, fetch, eviction, address, load instruction
13.7 The limitation of constant memory
This section explains why divergent constant-memory reads serialize, how cache locality affects their cost, and when global memory is preferable.
- Unique address serialization: Explain how a warp's constant-memory reads are served when its threads request different addresses, and apply the rule to repeated addresses.
Why it matters: Counting distinct addresses reveals the cost of a warp's constant-memory access. - Cache locality and serialization: Explain how requests for different addresses in the same cache line affect constant-memory rounds and latency.
Why it matters: Cache hits can make clustered scatter faster without eliminating serialization. - Memory choice for scatter: Justify leaving data in global memory when threads need different addresses, contrasting its handling of requests with constant memory.
Why it matters: Choosing memory for the access pattern can avoid the penalty of serialized constant-cache reads.
Key terms: constant cache, broadcast, unique addresses, serialized rounds, cache line, cache hit, scattered but clustered, global memory, scattered access, coalesce
13.8 Putting it all together: constant memory array scaling
This section applies the three-step constant-memory pattern to scaling an array and explains how to adapt the example for submission.
- Constant memory scaling pattern: Explain how the three-step constant memory pattern scales an array and why the shared factor suits broadcast.
Why it matters: This example shows how to use constant memory for a value shared by all threads. - Source declaration and ordering: Explain why the declaration and kernel appear before main() even though the copy step happens inside main().
Why it matters: The source order differs from the conceptual order of the three steps because names must be declared before use. - Submission input and formatting: Describe how to adapt the hard-coded example for the swforces problem, including its output formatting.
Why it matters: Correct input and output formatting lets the example work with the problem's supplied values and numeric comparison.
Key terms: constant memory, scaleFactor, cudaMemcpyToSymbol(), broadcast, warp, __constant__, kernel, main(), standard input, %g
13.9 Step 1: Declare in constant memory
This section explains how a file-scope declaration places scaleFactor in constant memory for cached, warp-wide access.
- Constant memory declaration: Describe where scaleFactor is declared and what the __constant__ qualifier does.
Why it matters: The declaration puts the value in the memory region that supports cached, warp-wide access.
Key terms: file scope, __constant__, constant memory, warp, broadcast
13.10 Step 2: Copy data to constant memory from the host
This section explains why host-side copying uses a constant-memory symbol rather than a dynamically allocated device pointer.
- Constant memory symbol lifetime: Explain why the host passes the constant-memory symbol name directly and why it does not free that memory.
Why it matters: This distinction helps you handle constant memory without treating it like dynamically allocated device memory.
Key terms: symbol name, scaleFactor, __constant__, statically allocated, cudaMalloc, cudaFree
13.11 Step 3: Use it in the kernel
This section traces an array-scaling kernel that broadcasts a constant scale factor and compares explicit constant memory with a scalar kernel argument.
- Constant memory scaling kernel: Trace how active threads scale separate array elements using a shared constant-memory value.
Why it matters: This shows how separate per-thread work can share one efficiently delivered read-only value. - Host constant data copy: Explain how copying the scale factor to constant memory differs from the familiar host workflow.
Why it matters: You can reuse the familiar GPU workflow while selecting the correct copy operation for the constant variable. - Scalar argument versus constant: Distinguish passing a uniform scalar as a kernel argument from explicitly declaring it in constant memory.
Why it matters: This helps you avoid adding explicit constant-memory setup when a scalar argument already provides the same broadcast.
Key terms: scaleFactor, constant memory, broadcast, global memory, idx, cudaMemcpyToSymbol, cudaMemcpy, constant variable, kernel argument, __constant__
13.12 Common mistakes
This section identifies three mistakes caused by treating constant memory like ordinary memory and states their consequences.
- Constant memory size limit: Explain what happens when an array exceeds the constant memory region.
Why it matters: Knowing the size limit helps you recognize why an oversized constant array fails. - Constant symbol copy mistake: Describe the mistake of passing a constant-memory symbol to cudaMemcpy as a device pointer and its stated consequence.
Why it matters: This mistake can produce computations using old data rather than the intended values. - Const qualifier distinction: Distinguish plain const from __constant__ in terms of the stated performance consequence.
Why it matters: A kernel can run without warning while missing the intended constant-memory performance.
Key terms: constant memory, 64 KB, __constant__, cudaMemcpy, device pointer, stale values, const, global-memory speed
13.13 Mistake 1: Exceeding 64 KB
This section explains how to recognize and avoid exceeding the GPU’s constant-memory capacity.
- Constant memory capacity limit: Explain why exceeding the constant-memory limit causes a build failure and how to avoid it.
Why it matters: Checking the combined size of constant data prevents a build failure and leaves space for other constants.
Key terms: constant memory, __constant__, 64 KB, ptxas
13.14 Mistake 2: Using the wrong copy function
This section explains why copying to a constant-memory symbol requires cudaMemcpyToSymbol and how to update part of a constant array.
- Constant Symbol Copy Errors: Explain why cudaMemcpy fails when used with a __constant__ symbol and how to copy the data correctly.
Why it matters: This prevents a copy that compiles from silently leaving the kernel with stale constant data. - Partial Constant Array Updates: Describe how cudaMemcpyToSymbol can update part of a constant array and how its default copy direction can be changed.
Why it matters: Partial updates avoid recopying constant data that has not changed between kernel launches.
Key terms: cudaMemcpy, __constant__, cudaMemcpyToSymbol, invalid-argument error, offset, cudaMemcpyHostToDevice, cudaMemcpyDeviceToDevice
13.15 Mistake 3: Using const instead of __constant__
This section distinguishes CUDA constant memory from ordinary const declarations and shows how to correct the mistake.
- Const versus constant memory: Distinguish a host const and a const pointer parameter from data placed in CUDA constant memory.
Why it matters: Read-only data does not automatically gain the constant memory cache and broadcast benefits. - Constant memory declaration and loading: Describe how to replace an ordinary const declaration with data in constant memory and identify the benefit of the fix.
Why it matters: This is how the data reaches the hardware region that provides constant-memory benefits. - Small lookup table example: Explain why the five-entry character lookup table can still work well in constant memory even when threads use different indices.
Why it matters: Different thread indices do not necessarily make a small constant-memory lookup table a poor choice.
Key terms: const, __constant__, constant memory, global memory, file scope, cudaMemcpyToSymbol(), constant region, broadcast, lookup table, cache line
13.16 Summary
Concepts explained in this chapter
- constant memoryConstant memory is a read-only CUDA memory space backed by device DRAM, with a limited address space and a dedicated cache near the streaming multiprocessors.
- constant cacheThe constant cache is a dedicated on-chip GPU cache that stores data read from constant memory for fast access, especially when threads request the same value.
- cudaMemcpyToSymbolcudaMemcpyToSymbol is a CUDA runtime API function that copies data to a named device symbol.
- __constant____constant__ is a CUDA variable qualifier that places a variable in the device's constant memory address space.
- const __restrict__ qualifiersA const __restrict__ pointer is a pointer declaration that marks its data as read-only and promises the referenced data is not aliased through another pointer.
- shared memory broadcastA memory broadcast supplies one value read from a single address to multiple threads.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
