Suppose two GPU threads add 50 and 30 to the same counter, initially zero. If both read zero before either writes, one update can overwrite the other: the result may be 50 or 30 instead of 80. A plain addition hides three steps—read, modify, and write—and another thread can slip in between them. section 5.6
An atomic operation makes the update indivisible with respect to competing updates. CUDA’s atomicAdd(result, value) updates the value at an address without losing another thread’s contribution. Updates to the same address take turns, though their order is unpredictable; updates to different addresses can proceed in parallel. section 5.15 section 5.6
Figure 5.8 Why atomicAdd is safe: read-modify-write as one indivisible step. When two blocks target the same address, the hardware serializes their requests: while one block's three steps (read → modify → write) run, the others wait, and the next one begins by reading the updated value, so no update is lost. When blocks target different addresses, their read-modify-writes run fully in parallel. The order in which same-address requests are served is unpredictable, but the final sum is always correct.

Compare competing updates to one address with independent updates to different addresses. section 5.15
Count into a histogram bin
If several threads use hist[bin]++, their increments can overwrite one another, producing counts that are too low or vary between runs. Replace the increment with atomicAdd(&hist[bin], 1). Pass the address of the bin, and initialize the histogram to zero on the device before counting. section 8.10 section 8.12
atomicAdd returns the value that was at the address before the addition. When threads each add one to a shared counter, those returned values are unique and can serve as output-slot indices. For a counter that is already a pointer, pass count rather than &count. section 8.3 section 8.12
Other CUDA atomics include subtraction, minimum, maximum, exchange, and compare-and-swap. Compare-and-swap writes a new value only if the current value matches an expected one; a retry loop can use the returned value to try again after a competing update. section 8.4
Table 8.1 The most useful CUDA atomic operations and their real-life analogies.
|
Operation
|
What it
does
|
Real-life
analogy
|
|
atomicAdd(addr,
val)
|
Adds val
to *addr
|
Adding
graded exams to the class total
|
|
atomicSub(addr,
val)
|
Subtracts val
from *addr
|
Removing a
miscounted exam from the total
|
|
atomicMax(addr,
val)
|
Sets *addr
to max(*addr,
val)
|
Updating
the top score on the board
|
|
atomicMin(addr,
val)
|
Sets *addr
to min(*addr,
val)
|
Updating
the lowest score on the board
|
|
atomicCAS(addr,
cmp, val) (compare-and-swap)
|
If *addr
== cmp,
sets *addr
to val;
returns old value of *addr
|
Edit only
if nothing changed
|
|
atomicExch(addr,
val) (exchange)
|
Sets *addr
to val;
returns the old value of *addr
|
Posting a
new score sheet and keeping the old one
|
See the CUDA atomic operations and the update each performs. section 8.4
Atomics ensure correctness, but many updates to one address create contention and serialize. In a reduction, threads can first combine values within each block, then issue one atomicAdd per block to the global result. In a histogram, each block can tally into a private shared-memory histogram before merging its bins globally; both the local tally and global merge still need atomic updates. section 5.15 section 8.10 section 8.12
An atomic update is not the same as a barrier. CUDA’s __syncthreads() coordinates threads within one block; atomicAdd can safely combine updates from different blocks. Depending on the problem, separating writers into phases or giving each location a single owner can also avoid a shared-update race. section 5.15 section 8.10
