liveBook · Manning

atomicAdd

As explained in Grokking GPU Programming

atomicAdd is a CUDA atomic operation that adds a value to a memory location without allowing concurrent updates to interfere.

When many GPU threads add to the same counter, a regular += can lose updates: two threads may read the same old value and then overwrite each other’s results. CUDA’s atomicAdd protects the update by making the read, addition, and write one indivisible operation. section 5.6 section 5.15

Call atomicAdd with a pointer to the location and the value to add. For example, atomicAdd(count, 1) increments a counter, while atomicAdd(&hist[bin], 1) increments a histogram bin. The & is needed because hist[bin] is a value, whereas count is already a pointer. Initialize counters and histogram bins before the kernel runs. section 8.3 section 8.7

The operation also returns the value stored before the addition. If threads each add 1 to a shared counter, each receives a unique old value that it can use as an output-array index. section 8.3

Combining block sums

After a tree reduction, each block has a subtotal in sdata[0]. Only thread 0 calls atomicAdd(result, sdata[0]) to contribute it to the global result. If two blocks contribute 36 and 24 to a result starting at zero, either update can go first; both are applied and the result is 60. A plain *result += sdata[0] can instead lose a contribution and produce a total that varies between runs. section 5.13 section 5.15 section 5.24

Figure 5.8 Why atomicAdd is safe: read-modify-write as one indivisible step. When two blocks target the same address, the hardware serializes their requests: while one block's three steps (read → modify → write) run, the others wait, and the next one begins by reading the updated value, so no update is lost. When blocks target different addresses, their read-modify-writes run fully in parallel. The order in which same-address requests are served is unpredictable, but the final sum is always correct.

Compare competing updates to one address with updates to different addresses. section 5.15

Updates to the same address must take turns, so heavy contention can make atomics slow. A reduction can reduce contention by combining values within each block and making one atomic call per block instead of one per element. In a histogram, many threads hitting one bin can similarly form a queue; a block-local histogram can reduce pressure on global bins. section 5.15 section 8.7 section 8.11

An atomic update is not a block-wide barrier. __syncthreads() coordinates threads within a block, while atomicAdd can safely combine contributions from different blocks at a shared location. section 5.15 section 5.14

How Grokking GPU Programming covers it

How deep it goes
Explains indivisible read-modify-write, pointer arguments, return values, contention, and the contrast with block barriers.
Examples
Uses positive-number counting, block subtotals of 36 and 24, and histogram bin increments.
What you can do after
Apply atomicAdd in CUDA kernels to protect shared additions and use its returned old value for unique slot indices.
How it connects
Connects race conditions with reduction, histograms, shared memory privatization, and block synchronization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.