chapter thirteen

13 Read-only data optimization

 

This chapter covers

  • Why reading the same value repeatedly from global memory wastes cache space
  • How constant memory’s dedicated cache avoids that waste
  • The three-step pattern for putting read-only data into constant memory
  • How a single broadcast serves all 32 threads of a warp in one fetch
  • Why threads that read different addresses are serialized and up to 32× slower

Imagine 32 students who all need the same one-page answer key. The classroom is smart: the first student fetches it from the storage room and leaves a copy on a shared shelf just outside the classroom door, and the next 31 students grab it from there, no trip to the storage room needed. That sounds efficient, until the shelf fills up and gets cleared, tossing out that answer key to make room for something else before anybody notices. Now the next student who needs it has to make the slow trip to the storage room all over again.

One value, 32 reads

How constant memory works

Constant memory within the hierarchy

The three steps for using constant memory

The broadcast mechanism

How broadcast works, step by step

The limitation of constant memory

Putting it all together: constant memory array scaling

Step 1: Declare in constant memory

Step 2: Copy data to constant memory from the host

Step 3: Use it in the kernel

Common mistakes

Mistake 1: Exceeding 64 KB

Mistake 2: Using the wrong copy function

Mistake 3: Using const instead of __constant__

Summary