liveBook · Manning

Race condition

As explained in Grokking GPU Programming

A race condition is a concurrency error in which unsynchronized reads and writes let one thread overwrite a value before another thread reads it.

Imagine two threads adding to the same total. Both read 0; one adds 50 and the other adds 30. If both write back, one update can overwrite the other, leaving 50 or 30 instead of 80. This is a race condition: the result depends on which thread reaches shared data first. section 5.6

An expression such as `result += value` looks like one operation, but it reads, adds, and writes. Another thread can intervene between those steps. Threads that only read the same data do not race; trouble begins when at least one writes. section 5.6 section 8.1

A reduction loses contributions

Suppose eight blocks each contribute 256 to a total. With an unprotected `+=`, overlapping updates can be lost, producing 256 or 512 instead of 2,048. Replacing the update with `atomicAdd` applies each contribution as one indivisible operation. section 5.24

A neighbor read sees changing input

In an in-place update, thread 4 writes `A[4]` while thread 5 reads it. Thread 5 may see either the old or the new value. Read from a separate input array and write to an output array instead; also guard the first element against reading `A[-1]`. section 4.14

Choose protection to match the dependency:

  • Use an atomic operation when several threads must update the same location, such as a histogram bin. Atomics prevent lost updates, though contention can make them expensive. section 8.10 section 8.11
  • Separate reads from writes when threads need neighboring input. Iterative stencils can alternate between two buffers; an in-place stencil cannot be fixed across blocks with `__syncthreads()`, which synchronizes only within a block. section 9.14
  • Separate dependent work into synchronized phases or redesign it so each location has a single owner. In a double-buffered scan, a barrier at the end of each round makes writes available before the next round reads them. section 8.10 section 6.8

Look for shared `++` or `+=` without protection, or a thread reading data another thread may overwrite. A total that changes between runs is a warning, but a race can also produce the same wrong answer—or even the correct answer—repeatedly. section 8.10 section 8.11

How Grokking GPU Programming covers it

How deep it goes
Traces read-modify-write and neighbor-read interleavings, then gives fixes and debugging checks.
Examples
Walks through simultaneous bank deposits, eight block sums, and neighboring array updates.
What you can do after
Diagnose/debug unprotected shared updates and neighbor reads, then select atomic operations, separate buffers, or synchronized phases.
How it connects
Connects races to atomic operations, reduction, histogram, stencil, scan, and synchronization barriers.

Also explained in

Compare how 25 books explain race condition →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.