liveBook · Manning

Atomic operation

As explained in Grokking GPU Programming

An atomic operation performs a read-modify-write or similar memory update as an indivisible action with respect to other threads.

Suppose two GPU threads add 50 and 30 to the same counter, initially zero. If both read zero before either writes, one update can overwrite the other: the result may be 50 or 30 instead of 80. A plain addition hides three steps—read, modify, and write—and another thread can slip in between them. section 5.6

An atomic operation makes the update indivisible with respect to competing updates. CUDA’s atomicAdd(result, value) updates the value at an address without losing another thread’s contribution. Updates to the same address take turns, though their order is unpredictable; updates to different addresses can proceed in parallel. section 5.15 section 5.6

Figure 5.8 Why atomicAdd is safe: read-modify-write as one indivisible step. When two blocks target the same address, the hardware serializes their requests: while one block's three steps (read → modify → write) run, the others wait, and the next one begins by reading the updated value, so no update is lost. When blocks target different addresses, their read-modify-writes run fully in parallel. The order in which same-address requests are served is unpredictable, but the final sum is always correct.

Compare competing updates to one address with independent updates to different addresses. section 5.15

Count into a histogram bin

If several threads use hist[bin]++, their increments can overwrite one another, producing counts that are too low or vary between runs. Replace the increment with atomicAdd(&hist[bin], 1). Pass the address of the bin, and initialize the histogram to zero on the device before counting. section 8.10 section 8.12

atomicAdd returns the value that was at the address before the addition. When threads each add one to a shared counter, those returned values are unique and can serve as output-slot indices. For a counter that is already a pointer, pass count rather than &count. section 8.3 section 8.12

Other CUDA atomics include subtraction, minimum, maximum, exchange, and compare-and-swap. Compare-and-swap writes a new value only if the current value matches an expected one; a retry loop can use the returned value to try again after a competing update. section 8.4

Table 8.1 The most useful CUDA atomic operations and their real-life analogies.
Operation
What it does
Real-life analogy
atomicAdd(addr, val)
Adds val to *addr
Adding graded exams to the class total
atomicSub(addr, val)
Subtracts val from *addr
Removing a miscounted exam from the total
atomicMax(addr, val)
Sets *addr to max(*addr, val)
Updating the top score on the board
atomicMin(addr, val)
Sets *addr to min(*addr, val)
Updating the lowest score on the board
atomicCAS(addr, cmp, val) (compare-and-swap)
If *addr == cmp, sets *addr to val; returns old value of *addr
Edit only if nothing changed
atomicExch(addr, val) (exchange)
Sets *addr to val; returns the old value of *addr
Posting a new score sheet and keeping the old one

See the CUDA atomic operations and the update each performs. section 8.4

Atomics ensure correctness, but many updates to one address create contention and serialize. In a reduction, threads can first combine values within each block, then issue one atomicAdd per block to the global result. In a histogram, each block can tally into a private shared-memory histogram before merging its bins globally; both the local tally and global merge still need atomic updates. section 5.15 section 8.10 section 8.12

An atomic update is not the same as a barrier. CUDA’s __syncthreads() coordinates threads within one block; atomicAdd can safely combine updates from different blocks. Depending on the problem, separating writers into phases or giving each location a single owner can also avoid a shared-update race. section 5.15 section 8.10

How Grokking GPU Programming covers it

How deep it goes
Explains read-modify-write races, same-address serialization, CUDA atomic operations, compare-and-swap retries, and contention-reduction procedures.
Examples
Illustrates lost bank deposits, positive-value counting, histogram bins, block subtotals, and a compare-and-swap-based float maximum.
What you can do after
Apply it via CUDA atomicAdd to shared counters and histogram bins, and use compare-and-swap in a retry loop.
How it connects
Connects atomic operations to race conditions, barriers, reduction, histograms, shared memory, and privatization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.