10 Background Processing

 

This chapter covers

  • How to process background tasks in Kubernetes
  • The Kubernetes Job and CronJob objects
  • When to use (and not use) Job objects for your own batch processing workloads
  • Creating a custom task queue with Redis
  • Implementing a background processing task queue with Kubernetes

In the prior chapters we looked at developing services that are exposed on an IP address, whether it’s an providing an external service on a public address, or an internal service on a cluster local IP. But what about all the other computation that you may need to do that isn’t directly part of a request-response chain, like resizing a bunch of images, sending out device notifications, processing financial data, or rendering a movie one frame at a time? Background tasks are all the compute processes that take an input and produce an output without being part of the synchronous processing of requests in the way services are.

You can process background tasks using Deployment, or the Kubernetes Job object. Deployment is ideal for a continuously running task queue like the one most web applications run for tasks like image resizing. The Kubernetes Job construct is great for running one-off maintenance tasks, periodic tasks (via CronJob), and processing a batch workload when there is a set amount of work to complete.

10.1 Background Processing Queues

 
 
 

10.1.1 Creating a custom task queue

 
 
 
 

10.1.2 Signal Handling in Worker Pods

 
 

10.1.3 Scaling Worker Pods

 
 
 

10.1.4 Open Source Task Queues

 
 
 

10.2 Jobs

 

10.2.1 Running one-off tasks with Jobs

 
 

10.2.2 Scheduling Tasks with Cron Jobs

 
 

10.3 Batch task processing with Jobs

 
 
 
 

10.3.1 Dynamic Queue Processing with Jobs

 

10.3.2 Static Queue Processing with Jobs

 
 
 
 

10.4 Liveness Probes for Background Tasks

 
 
 

10.5 Summary

 
 
 
sitemap

Unable to load book!

The book could not be loaded.

(try again in a couple of minutes)

manning.com homepage
test yourself with a liveTest