A worker pool runs a fixed number of goroutines that take work from a shared channel. A common mistake is choosing the pool size before asking what limits the work: CPU speed, I/O speed, or available memory. section 8.5
Processing chunks from a reader
Read 1,024-byte chunks from an io.Reader, send each chunk to the workers, and sum the results of task(b). The sequential version calls task after every read; the pooled version lets workers process chunks concurrently. section 8.5
func read(r io.Reader) (int, error) {
var count int64
wg := sync.WaitGroup{}
var n = 10
ch := make(chan []byte, n) #1
wg.Add(n) #2
for i := 0; i < n; i++ { #3
go func() {
defer wg.Done() #4
for b := range ch { #5
v := task(b)
atomic.AddInt64(&count, int64(v))
}
}()
}
for {
b := make([]byte, 1024)
// Read from r to b
ch <- b #6
}
close(ch)
wg.Wait() #7
return int(count), nil
}
The example starts ten workers, gives the task channel capacity ten, and uses a wait group and atomic additions to collect results. Its read loop is a sketch: the read operation and termination condition still need implementation before close(ch) can be reached. section 8.5
Choose the pool and its channel deliberately:
- Classify the task first. A CPU-bound task is limited by CPU speed; an I/O-bound task waits on operations such as REST calls or database queries. GOMAXPROCS is shown as a possible pool-size value, not a universal size for every workload. section 8.5
- Buffering the task channel to the number of workers can reduce contention while the producer publishes tasks. Do not treat an unexplained channel capacity as a tuning rule: smaller queues can increase contention, while larger ones use more memory. section 8.5 section 9.7
