Concurrency structures a program as independent activities that make progress during overlapping periods. In Go, goroutines run concurrent functions, and channels communicate and synchronize their work through explicit message flow. section 1.4 section 1.4.1
Start work with goroutines
Prefixing a function call with go schedules it to run in a goroutine, so two crawl calls can proceed concurrently. Goroutines use the same function syntax as sequential code and are lightweight enough that thread pools are usually unnecessary, although repeatedly creating goroutines for tiny, frequent tasks can add overhead. section 1.4.1
Figure 1.3 The go statement before a function spawns a new goroutine that runs the given function concurrently. This figure illustrates running the crawl function concurrently in two separate goroutines.
The diagram shows a go statement spawning separate goroutines for two crawl operations. section 1.4.1
The runtime scheduler multiplexes goroutines onto a smaller set of threads. When a goroutine blocks on a channel or I/O, it is parked so a ready goroutine can run; nonblocking I/O helps avoid blocking operating-system threads for I/O-bound work. Concurrency is distinct from parallelism: multiple CPU cores may run threads in parallel, but concurrent structure can also execute sequentially. section 1.4.2 section 6.9.3
A concurrent pipeline organizes work into stages:
- Connect focused stages with channels, passing messages from producers through intermediate work to final consumers. section 6.6.1
- Add, remove, and reuse stages independently, improving flexibility, modularity, and maintainability. section 6.6.1
- Keep a package API synchronous when concurrency is an internal implementation choice; callers can still run synchronous functions concurrently themselves. section 6.1.2
Load-testing with a concurrent pipeline
A request client can retain its synchronous SendN API while internally piping requests through concurrent stages. With 100 requests that each take 100 ms, one worker finishes in about 10 seconds, while 10 workers finish in just over 1 second; a 25-RPS throttle limits that run to about 4 seconds. Raising concurrency eventually has diminishing returns when resources or nonparallelizable work become bottlenecks. section 6.6 section 6.9.3
