2 Your first parallel program
This chapter covers
- The structure of a parallel CUDA program
- How to write a kernel function with __global__
- How to launch parallel work with <<<...>>>
- How threads use threadIdx to identify themselves
- How the hardware runs your threads: warps, SMs, and CUDA cores
- How to transfer data between the host (CPU) and the device (GPU)
You met the grading room in chapter 1: a stack of 1,000 exams and 1,000 students who could each grade one. In this chapter, we’ll put that analogy to use as a guide to developing a working program. By the end, you’ll have written the “grade the exam in front of you” instruction in CUDA and submitted solutions to this chapter’s problems on swforces.com. All you need on hand is a swforces.com account and some basic C: functions, loops, and a little comfort with pointers, because malloc and free will make a brief appearance.
Hello, Parallel World!
Every programmer starts with “Hello, World.” Ours has a twist: instead of printing the message once, we’ll have eight threads print it at the same time. Picture those 1,000 students again: each student is one thread, a single worker that runs your code on its own small slice of the work. A parallel program launches many threads at once, and every one of them runs the same code independently.