To double an array, a sequential loop handles one element at a time. A parallel approach can give each element its own worker. That worker is a thread: a sequence of instructions that performs a unit of computation. In CUDA, many threads execute the same kernel code independently, typically on different pieces of data. section 1.4 section 2.3
#include <stdio.h>
#include <cuda_runtime.h>
__global__ void helloKernel() { #A
printf("Hello, Parallel World!\n"); #B
}
int main() {
helloKernel<<<1, 8>>>(); #C
cudaDeviceSynchronize(); #D
return 0;
}The launch creates eight threads. Each runs the kernel’s print statement once; cudaDeviceSynchronize() makes the host wait for completion. section 2.1
Think of a grid as a school, a block as a classroom, and a thread as a student doing the work. Each kernel launch creates a grid of blocks, and each thread belongs to one block. For example, a launch with four blocks of 256 threads creates 1,024 threads in total. Blocks can run in any order, so independent work must not rely on one block finishing before another. section 3.2
A thread needs an index to find its own work. Within one block, threadIdx.x identifies its position; across multiple blocks, use idx = blockIdx.x * blockDim.x + threadIdx.x. Round the block count up with (n + threadsPerBlock - 1) / threadsPerBlock, then guard the operation with if (idx < n). The extra threads created by rounding up must not access elements beyond the array. section 3.16 section 2.24
One addition per thread
For vector addition, thread 0 computes C[0] = A[0] + B[0], and thread 1 computes C[1] = A[1] + B[1]. Each thread finds its global index, checks the bounds, and performs the same addition on its own elements. section 4.6
A thread’s unit of work need not be a single arithmetic operation. In matrix multiplication, one thread can compute one output element by accumulating a row–column dot product. With a two-dimensional launch, check that x maps to columns and y maps to rows, and guard both coordinates against the matrix bounds. section 12.1
Many threads do not imply that every thread runs at the same instant. Hardware runs threads in groups of 32 called warps, and extra blocks wait for a streaming multiprocessor to become available. Launching work and moving data also take time, so the ideal parallel step count is not a guaranteed speedup. section 1.4 section 3.2
A common launch mistake is putting more than 1,024 threads in one block: the launch fails and no threads run. Check cudaGetLastError() after a launch to find the error, and use multiple blocks for larger workloads. Kernel launches return before completion; synchronize before relying on results when no other operation provides the necessary wait. section 3.16 section 4.6
















