liveBook · Manning

GPU memory

As explained in Grokking GPU Programming

Device memory is memory located on a GPU and accessible to GPU computations.

GPU device memory is storage on the GPU that GPU computations can access. A kernel’s pointer must refer to device memory, not regular CPU (host) memory. CUDA allocates device memory with cudaMalloc and transfers data with cudaMemcpy. section 2.10 section 3.6

Global memory is the large, widely accessible device-memory tier: threads across the grid can reach it, but it is slower than shared memory. Tiling speeds matrix multiplication by loading data from global memory into shared memory, where threads in a block can reuse it. section 3.6 section 12.2

A typical CUDA program handles device memory in this order:

  • Allocate GPU memory, then copy input data from host memory to the device. section 2.24
  • Launch the kernel with device-memory pointers; synchronize before relying on its results. section 2.10 section 2.24
  • Copy results back to the host and free the GPU memory. section 2.24

A pointer from the wrong side

Passing a regular host-memory pointer to a kernel that expects device memory is a mistake. Allocate the array on the GPU and transfer its contents before using it in the kernel. section 2.10 section 2.24

How Grokking GPU Programming covers it

How deep it goes
Defines device and global memory, outlines allocation and transfers, and contrasts global with shared memory without detailing hardware internals.
Examples
Uses an integer array passed to a kernel and matrix multiplication that reuses global-memory data through tiling.
What you can do after
Operate on GPU data by allocating device memory, transferring inputs and results, and passing device pointers to kernels.
How it connects
Links device memory to host memory, CUDA kernels, global memory, shared memory, and data transfer.

Also explained in

Compare how 6 books explain GPU memory →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking GPU Programming. The section links point to the parts of the book it draws on.