A worker is a background service that consumes queued work independently of request-serving services. A channel can serve as the queue, while one or more goroutines ingest feeds without blocking the API. section 9.5.1
A worker pool organizes concurrent processing around a shared queue:
- Create workers that read feed URLs from an incoming channel and process them sequentially. section 9.5.1
- Share a buffered channel across workers so submitted URLs go to the next available worker. section 9.5.1
- Configure pool size at creation time to increase concurrent throughput as load grows. section 9.5.1
- Stop the pool by closing its incoming channel and integrate lifecycle management into the application. section 9.5.1
Separate feed ingestion from the API
Run the worker as one process and the API as another. The worker fetches configured feeds every five minutes and stores articles; the API serves the stored, aggregated results without performing network fetches. A failed or malformed feed is logged while remaining feeds continue processing. section 11.3.4
Figure 9.2 Worker pools ingest incoming messages across multiple processes.
Visualizes how a worker pool distributes incoming messages across concurrent processing units. section 9.5.1
