liveBook · Manning

Constant cache

As explained in Grokking GPU Programming

The constant cache is a dedicated on-chip GPU cache that stores data read from constant memory for fast access, especially when threads request the same value.

Imagine many GPU threads scaling pixels by the same factor. Each thread needs the same value, so fetching it separately would repeat work. The constant cache lets threads share that read: when a warp requests one constant-memory address, the value can be broadcast to all participating threads. section 13.5 section 13.6

Constant memory is a 64 KB read-only address space in device DRAM. Its dedicated on-chip cache sits near each streaming multiprocessor (SM), the GPU’s processing unit. The 64 KB limit describes the address space, not the size of each SM’s cache. Device threads read constants; the host supplies their values. section 13.3 section 13.16

Figure 13.2 The GPU memory hierarchy as a classroom: each thread (student) has its own registers (sticky notes on the desk); threads in a block share that block’s shared memory (whiteboard); the per-SM L1 and constant caches and the GPU-wide L2 cache act as shelves; and device DRAM (the storage room) holds both global memory (the filing cabinet) and constant memory (the blackboard).

The classroom diagram places the constant cache near the processors and its backing constant memory in device DRAM. section 13.3

A warp is a group of 32 threads that issues a load together. If its active threads request the same address, one transaction serves them all; inactive threads do not participate. The cache can still miss, such as on a first read or when one constant evicts another. Dedicated broadcast means per-thread data cannot evict a constant from this cache, not that every read is a cache hit. section 13.6 section 13.5

Scaling pixels

Suppose every active thread multiplies its pixel by scaleFactor. Because they all read the same constant-memory address, the cache broadcasts one result across the warp. If the value were a three-float vector instead, each float would require a separate load and broadcast, though later loads could hit the same cache line. section 13.6

Different addresses change the picture. The cache serves one unique address per round, broadcasting each result to the threads that requested it. A warp requesting N distinct addresses needs N serialized rounds, up to 32. Addresses in the same cache line still require separate rounds, but after the line is fetched, later rounds can hit in the cache. section 13.7

A lookup-table trap

If one thread reads lookup[42], another reads lookup[7], and others choose their own indices, the reads may serialize even though the table is read-only. For a large read-only array or per-thread indexing, consider a const __restrict__ pointer instead of relying on constant-memory broadcast. section 13.7 section 13.16

To use constant memory in CUDA:

  • Declare a device symbol with __constant__, then copy host values to it with cudaMemcpyToSymbol(). No cudaMalloc or device pointer is needed for the symbol. section 13.16 section 13.4
  • Read the symbol in the kernel. Prefer a small array whose elements are read uniformly across a warp; for a single scalar, pass a kernel argument. section 13.16
  • Do not confuse __constant__ with ordinary const: a const device pointer still refers to global memory and does not provide the dedicated constant cache or its broadcast. section 13.4

How Grokking GPU Programming covers it

How deep it goes
Explains broadcast and serialized access step by step, including cache misses, active threads, and cache-line locality.
Examples
Uses pixel scaling with a shared factor and a lookup table indexed differently by individual threads.
What it assumes
Assumes familiarity with global memory, shared memory, and streaming multiprocessors while explaining broadcast through examples.
What you can do after
Apply it in CUDA by declaring and copying constants, then choosing access patterns suited to broadcast.
How it connects
Connects constant memory with GPU memory hierarchy, streaming multiprocessors, warps, cache lines, and global memory.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.