10 Background Processing
This chapter covers
- How to process background tasks in Kubernetes
- The Kubernetes Job and CronJob objects
- When to use (and not use) Job objects for your own batch processing workloads
- Creating a custom task queue with Redis
- Implementing a background processing task queue with Kubernetes
In the prior chapters we looked at developing services that are exposed on an IP address, whether it’s an providing an external service on a public address, or an internal service on a cluster local IP. But what about all the other computation that you may need to do that isn’t directly part of a request-response chain, like resizing a bunch of images, sending out device notifications, processing financial data, or rendering a movie one frame at a time? Background tasks are all the compute processes that take an input and produce an output without being part of the synchronous processing of requests in the way services are.
You can process background tasks using Deployment, or the Kubernetes Job object. Deployment is ideal for a continuously running task queue like the one most web applications run for tasks like image resizing. The Kubernetes Job construct is great for running one-off maintenance tasks, periodic tasks (via CronJob), and processing a batch workload when there is a set amount of work to complete.