liveBook · Manning

Coarse parallel processing

As explained in Kubernetes in Action, Second Edition

Coarse parallel processing assigns one work item to each pod, with each pod terminating after processing that item.

Coarse parallel processing assigns one work item from a queue to each pod. After processing that item, the pod terminates, producing roughly one pod per work item. section 18.1.5

The queue and its worker logic belong in the application containers; Kubernetes runs those containers in one or more Job pods. Compared with fine parallel processing, coarse processing uses short-lived pods that handle single items, while fine processing lets a smaller group of pods handle multiple items. section 18.1.5

Processing queued months

A MongoDB collection stores twelve monthly work items. A Kubernetes Job can set completions to 6 and parallelism to 3; each worker script removes one item with findOneAndDelete(), processes it, prints completion, and exits. section 18.1.5

Figure 18.10 The difference between coarse and fine parallel processing
figure

Compare how coarse processing creates one terminating pod per item with fine processing, where a smaller pod group drains the queue. section 18.1.5

How Kubernetes in Action, Second Edition covers it

How deep it goes
Explains pod-per-item processing, contrasts fine processing, and demonstrates Job and worker-script configuration.
Examples
Uses MongoDB monthly items, a six-completion Job with parallelism three, and a single-item mongosh worker script.
What it assumes
Assumes an application-provided queue and worker container, while introducing how Kubernetes Jobs run those workers.
What you can do after
Builds a queue-backed Kubernetes Job whose worker container claims and processes one item before terminating.
How it connects
Connects work queues with Jobs, pods, containers, MongoDB, and fine parallel processing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kubernetes in Action, Second Edition. The section links point to the parts of the book it draws on.