A work queue holds pending work items that workers retrieve and process. Unlike static Job input, the work is assigned dynamically through a queue. Kubernetes does not provide the queue or its retrieval logic; containers implement both, while a Job runs those containers in one or more pods. section 18.1.5
Choose a processing shape based on how workers should consume the queue:
- Coarse parallel processing creates one pod per work item. Each pod retrieves one item, processes it, and terminates. section 18.1.5
- Fine parallel processing creates a smaller group of pods. Each pod repeatedly retrieves and processes multiple items until the queue is empty. section 18.1.5
- Both approaches can run workers in parallel, subject to the capacity of the cluster. section 18.1.5
MongoDB-backed queue
A queue can be represented by the MongoDB collection monthsToProcess. A worker atomically retrieves and removes one document with findOneAndDelete({}); if no document remains, it exits successfully. A fine-grained worker repeats this operation until the collection is empty.
Figure 18.10 The difference between coarse and fine parallel processing
Compare coarse and fine parallel processing by worker lifetime, item assignment, and number of pods. section 18.1.5
