10 Background processing

 

This chapter covers

  • How to process background tasks in Kubernetes
  • The Kubernetes Job and CronJob objects
  • When to use (and not use) Job objects for your own batch processing workloads
  • Creating a custom task queue with Redis
  • Implementing a background processing task queue with Kubernetes

In prior chapters, we looked at developing services that are exposed on an IP address, whether it’s providing an external service on a public address or an internal service on a cluster local IP. But what about all the other computations that you may need to do that aren’t directly part of a request-response chain, like resizing a bunch of images, sending out device notifications, running an AI/ML training job, processing financial data, or rendering a movie one frame at a time? These are typically processed as background tasks, which are processes that take an input and produce an output without being part of the synchronous handling of user requests.

You can process background tasks using Deployment or the Kubernetes Job object. Deployment is ideal for a continuously running task queue like the one most web applications run for tasks such as image resizing. The Kubernetes Job construct is great for running one-off maintenance tasks, periodic tasks (via CronJob), and processing a batch workload when there is a set amount of work to complete.

10.1 Background processing queues

10.1.1 Creating a custom task queue

10.1.2 Signal handling in worker Pods

10.1.3 Scaling worker Pods

10.1.4 Open source task queues

10.2 Jobs

10.2.1 Running one-off tasks with Jobs

10.2.2 Scheduling tasks with CronJobs

10.3 Batch task processing with Jobs

10.3.1 Dynamic queue processing with Jobs

10.3.2 Static queue processing with Jobs

10.4 Liveness probes for background tasks

Summary

sitemap