Chapter 3: Building applications on Kubernetes

Introduction

This chapter introduces the infrastructure tools and practices that support reliable ML systems, from packaging applications to deploying, automating, and monitoring them.

• Infrastructure foundation for ML systems: Explain why ML engineers need infrastructure skills: deploying models, running pipelines, and maintaining an ML platform depend on a solid foundation of DevOps tools and practices, even though those tools are not specific to ML.
  Why it matters: This connects the chapter’s infrastructure work to the ML systems you will build later.
• Application infrastructure tool roadmap: Describe the chapter’s progression from packaging applications consistently with Docker, to deploying and managing them with Kubernetes, automating builds and deployments with CI/CD, and monitoring application health and performance.
  Why it matters: This sequence shows how the chapter moves from an application package to an operated production application.

Key terms: ML platform, ML engineer, DevOps tools and practices, infrastructure, Docker, Kubernetes, continuous integration/continuous deployment (CI/CD), monitoring

3.1 Containers and tooling

The section introduces the tools and workflow for packaging, deploying, automating, and monitoring a sample application on Kubernetes.

• Container packaging and orchestration: Explain that Docker packages an application with its libraries and runtime into a container, while Kubernetes deploys containers, manages their resources, scales them under load, and replaces failed ones. Describe why Kubernetes is useful across on-premises and cloud environments.
  Why it matters: Packaging and orchestration make model-serving applications more consistent and resilient across environments.
• Deployment automation and monitoring: Describe how GitLab CI and Argo CD are used to automate CI/CD deployments, while Prometheus tracks application metrics and Grafana visualizes them. Explain that automation and monitoring help identify and address application problems before they escalate.
  Why it matters: A deployed application also needs repeatable releases and visibility into how it is behaving.
• Sample application chapter workflow: Trace the chapter’s plan for a FastAPI application whose endpoint displays a random joke: containerize it, deploy it to a container orchestrator, then add automation and monitoring. State that the named tools are choices toward an automated ML platform, not mandatory tools for every organization.
  Why it matters: The small application provides a concrete way to practice the deployment workflow before applying it to ML workloads.

Key terms: Docker, containerization, Kubernetes, resource management, scaling, cloud agnostic, continuous integration (CI), continuous deployment (CD), GitLab CI, Argo CD

3.2 Docker

This section introduces Docker containerization as a way to package an application with its dependencies so it can run consistently across local and production environments.

• Dependency packaging and portability: Explain why deploying a locally developed application can fail when the cloud environment lacks its libraries or uses a different Linux distribution, and how Docker addresses this by packaging the application with its dependencies in a container.
  Why it matters: Packaging dependencies with the application reduces environment-related deployment problems.
• Shipping container analogy: Describe how standardized shipping containers illustrate Docker containers: applications are the goods, dependencies and configurations are the packaging and handling requirements, and containers make varied applications easier to move and run across environments.
  Why it matters: The analogy makes clear why consistent packaging simplifies application delivery.

Key terms: Docker, containerization, container, dependencies, shipping containers, Docker containers, applications or services, configurations

3.2.1 Write the application code

This section creates a small FastAPI joke application, checks that it runs locally, and prepares to containerize it with Docker.

• FastAPI application and files: Describe the example as a Python FastAPI application that displays a random joke, with source code in main.py and dependencies in requirements.txt. Explain that FastAPI can also be used to build APIs that serve ML models.
  Why it matters: The small joke app provides a working example before the book moves to ML-serving applications.
• Local execution and Docker preparation: Explain the sequence of installing the requirements, running the FastAPI app in main.py with Uvicorn on port 8083, and using curl to check the endpoint before containerizing it. State that Docker Desktop must be installed to build the Docker container.
  Why it matters: Checking the app locally establishes that it works before Docker packaging adds another layer to troubleshoot.

Key terms: FastAPI, main.py, requirements.txt, API, Uvicorn, ASGI server, port 8083, curl, Docker Desktop

3.2.2 Write a Dockerfile

This section explains how a Dockerfile builds the FastAPI application image and specifies the script that runs when the container starts.

• Dockerfile build instructions: Trace how FROM establishes a base image, WORKDIR sets the app directory, RUN executes setup commands, and COPY brings in Python requirements and application files.
  Why it matters: These instructions explain how the application and its dependencies become part of the image.
• Build argument and environment variable: Explain that ARG declares an argument passed at build time and that ENV uses it here to set the ENVIRONMENT environment variable.
  Why it matters: This distinguishes a build-time input from the environment variable set in the image.
• Container startup command: Describe how the Dockerfile makes entrypoint.sh executable and uses ENTRYPOINT to define the default command run when the image is started as a container.
  Why it matters: Building the image is not enough; the container also needs a command to run the application.

Key terms: Dockerfile, FROM, base image, layer, WORKDIR, RUN, ARG, build argument, ENV, ENVIRONMENT

3.2.3 Building and pushing a Docker image

This section builds a Docker image from a Dockerfile, tests it as a local container, and pushes a tagged copy to a registry for deployment.

• Docker image build and tags: Explain that building from a Dockerfile produces an image containing the application and what it needs to run, and describe how a name and tag identify an image version or variant.
  Why it matters: The image is the package you can test locally and later deploy elsewhere.
• Local container test and ports: Distinguish an image from the container created when it is run, and describe testing the application locally with docker run and a published host port.
  Why it matters: Running the image locally checks the packaged application before it is pushed for deployment.
• Registry push and retrieval: Trace how a locally tested image is stored in Docker Hub: create a repository, log in, retag the image for that repository, and push it. Explain that a container registry stores image versions for an orchestrator such as Kubernetes to pull and run.
  Why it matters: A registry makes the tested image available to the system that will deploy it.

Key terms: Dockerfile, Docker image, repository, tag, image ID, docker run, container, -p, -it, localhost:8081

3.3 Kubernetes

This section introduces Kubernetes as the tool that schedules Docker containers on servers and brings them back up when they fail outside the development environment.

• Kubernetes container orchestration role: Explain why running a container with `docker run` is not enough for deployment beyond development: containers need to be scheduled on servers and brought back up after failures such as out-of-memory errors, tasks Kubernetes handles.
  Why it matters: This is the reason the chapter moves from Docker containers to Kubernetes.

Key terms: Kubernetes, K8s, container orchestration, docker run, Out of Memory (OOM)

3.3.1 Kubernetes architecture overview

This section explains why Kubernetes uses a multi-node cluster and how master and worker services coordinate to run containers.

• Multi-node cluster purpose: Explain why a single node is a risk and how a cluster with worker nodes and a master node supports resilience and scaling.
  Why it matters: The reason for multiple nodes makes Kubernetes orchestration easier to understand.
• Master-node orchestration services: Describe the four master-node services: the API server accepts commands, etcd stores cluster state, the scheduler selects a suitable node using resource needs and availability, and the controller responds when a container goes down.
  Why it matters: These distinct services explain how a request becomes a placement decision and how failures are handled.
• Worker runtime and kubelet: Describe how the container runtime runs application containers on a worker node and how the kubelet reports node and container health to the master and carries out its requested actions.
  Why it matters: The worker-side services connect the master's decisions to containers actually running on nodes.

Key terms: cluster, master node, worker nodes, resilience, scaling, API server, etcd key-value store, scheduler, controller, CPU

3.3.2 Kubectl

Introduces kubectl as the command-line tool for inspecting and managing a Kubernetes cluster, with examples of node inspection, cluster information, and application deployment.

• Kubectl cluster management role: Explain that kubectl is the command-line utility for interacting with a Kubernetes cluster, including inspecting the cluster and deploying applications.
  Why it matters: Kubectl is the tool used to carry out and check Kubernetes operations from the command line.
• Node inspection output: Describe the information shown when kubectl lists cluster nodes: each node’s name, readiness status, role or label, age, and Kubernetes version.
  Why it matters: Those fields let you identify the nodes and see basic information about their state.

Key terms: kubectl, K8s cluster, nodes, status, role, node label, age, K8s version

3.3.3 Kubernetes objects

The section builds a FastAPI application on Kubernetes from a pod through a ReplicaSet to a Deployment, showing how to inspect, scale, update, and recover it.

• Pod structure and definition: Explain that a pod is Kubernetes’ smallest execution unit: one application instance that can contain multiple containers sharing network and storage. Describe the four main YAML keys and what the pod spec supplies for the hello-joker application.
  Why it matters: Pods are the building blocks managed by the other Kubernetes objects in this section.
• Pod inspection and testing: Describe how the section creates or updates a pod, checks its readiness and status, investigates an image-pull error, reads logs, tests the application through port forwarding, and deletes the pod.
  Why it matters: These commands let you check whether an application actually started and diagnose why it did not.
• ReplicaSet desired replica count: Explain how a ReplicaSet uses a label selector to identify pods, a replica count to specify the desired number, and a pod template to create replacements or additional pods. Describe its response to deletion, scaling, and a matching pod that existed before the ReplicaSet.
  Why it matters: A ReplicaSet maintains the requested number of application pods when conditions change.
• Deployment updates and rollback: Distinguish a Deployment from a ReplicaSet by explaining that a Deployment manages ReplicaSets, which manage pods, and adds gradual image updates and rollback. Trace why a failed image update stops progressing without taking down the three working old-version pods, and how the section undoes it.
  Why it matters: Deployments provide the update and recovery behavior used for the later stateless applications and inference services.

Key terms: pod, apiVersion, kind, metadata, spec, container image, kubectl create, kubectl apply, kubectl get, READY

3.3.4 Networking and services

Kubernetes Services give changing pods stable ways to communicate within a cluster or receive traffic from outside it, using NodePort, ClusterIP, or LoadBalancer access.

• Stable access to changing pods: Explain why applications and users access pods through Services rather than relying on pod IP addresses or kubectl port-forward. Describe how a Service uses a selector to direct traffic to matching pods.
  Why it matters: This explains how an application remains reachable when its pods are replaced or moved.
• NodePort external access path: Trace a request to a NodePort Service from a node IP and nodePort, through the Service port, to a pod’s targetPort. Explain why any node’s IP can be used even if the pod runs elsewhere.
  Why it matters: This gives you a concrete way to reach and test an application from outside the cluster.
• ClusterIP internal service access: Describe how ClusterIP enables communication between applications inside the cluster without exposing the Service externally. Explain that another pod can use the Service’s internal IP and port or its DNS-resolved Service name, while the Service routes to matching pods.
  Why it matters: This is the access pattern for components such as an in-cluster frontend calling a backend.
• LoadBalancer public access path: Explain how a LoadBalancer Service exposes selected pods through an external load balancer and externally assigned IP address. Distinguish that access path from NodePort’s node-IP-and-port access and ClusterIP’s internal-only access.
  Why it matters: The three access paths help you choose how an application or API should be reached.

Key terms: Service, pod IP address, kubectl port-forward, selector, labels, NodePort, port, targetPort, node IP, ClusterIP

3.3.5 Other objects

This section introduces namespaces, ConfigMaps, and Secrets as Kubernetes objects for organizing resources and supplying application configuration.

• Namespace isolation and defaults: Explain how namespaces organize resources within one physical cluster, separating teams or applications and avoiding naming conflicts. State that resources go into the default namespace when none is specified.
  Why it matters: Namespaces help keep a shared cluster manageable as applications grow.
• Namespace placement and service access: Describe how the section creates and inspects a namespace, places a deployment and service in it through their metadata, and accesses a service from another namespace using a namespace-qualified DNS name.
  Why it matters: Creating a namespace is useful only if you can place resources there and reach their services.
• ConfigMap configuration separation: Explain that a ConfigMap holds non-sensitive configuration separately from application code or containers, and describe its named metadata and key-value data.
  Why it matters: Separating configuration makes settings easier to manage without changing application code.
• Secret handling and encoding limits: Describe Secrets as objects for sensitive values and distinguish base64 encoding from encryption. Explain that the section recommends encryption or dedicated secret management for genuinely protecting sensitive information.
  Why it matters: Mistaking readable, encoded credentials for encrypted ones can expose them.
• Pod environment variable references: Trace how a pod obtains environment-variable values from a ConfigMap or Secret using env, valueFrom, and the appropriate key reference.
  Why it matters: These references connect stored configuration and credentials to an application running in a pod.

Key terms: namespace, default namespace, kubectl create namespace, -n, metadata, ClusterIP, DNS, ConfigMap, data, key-value pairs

3.3.6 Helm charts

This section introduces Helm charts as a way to package, configure, install, and manage applications made of multiple Kubernetes objects.

• Helm chart packaging: Explain why managing an application as separate Kubernetes objects is cumbersome and how Helm packages their templates and configurable values into a chart.
  Why it matters: This explains why one chart can manage an application that needs several Kubernetes objects.
• Chart repository and release workflow: Trace the Redis example from finding a chart in a repository through installation, inspection, and removal, distinguishing a chart from an installed Helm release.
  Why it matters: The Redis walkthrough shows the basic commands for using an existing chart rather than writing one.
• Chart inspection and customization: Describe how to pull and inspect a chart, then create and configure the hello-joker chart so one Helm installation produces a three-replica Deployment and a NodePort Service.
  Why it matters: This connects chart files and values to the Kubernetes objects created by an installation.

Key terms: Helm, chart, object templates, values.yaml, K8s objects, Artifact Hub, chart repository, helm repo update, helm install, release name

3.3.7 Conclusion

This conclusion recaps how Docker and Kubernetes provide a starting point for deploying applications and why Kubernetes suits resource-intensive ML workloads.

• Docker and Kubernetes roles: Distinguish Docker’s role in containerizing applications from Kubernetes’s role in orchestrating containers and automating deployments, including complicated applications.
  Why it matters: The distinction connects packaging an application with running and managing it.
• ML workload resource scaling: Explain why Kubernetes is well suited to dynamic, resource-intensive ML workloads: it helps scale and manage computing resources for training and inference, simplifying deployment and management so teams can focus on models.
  Why it matters: Resource demands are a central reason this book uses Kubernetes for ML applications.

Key terms: Docker, K8s, container orchestrator, ML workloads, scalability, resource efficiency, training, inferring

3.4 Continuous integration and deployment

This section explains why Kubernetes application releases should be automated and how GitLab CI and Argo CD divide the work of building, deploying, and keeping a release current.

• Reasons to automate deployments: Explain why manually building images and deploying frequently changing applications is risky, and why CI/CD is useful for ML applications as well as other software.
  Why it matters: Automation reduces the chance that a manual release puts the wrong version into production or disrupts a working deployment.
• CI and CD responsibilities: Distinguish the CI job that tests, builds, and updates a Helm chart from the CD job that deploys the chart with the newly built image tag. Describe how CD monitors the deployment and corrects an unintended rollback to an older image, using GitLab CI and Argo CD as the respective tools.
  Why it matters: The division of work explains how a newly built image becomes—and remains—the intended deployed version.

Key terms: Continuous Integration (CI), Continuous Deployment (CD), image tag, CI job, CD job, Helm chart, GitLab CI, Argo CD

3.4.1 GitLab CI

This section sets up GitLab CI to test a FastAPI application, publish a commit-tagged Docker image, and update a Helm chart for a separate CD tool to deploy.

• GitLab CI pipeline stages: Explain how the .gitlab-ci.yaml pipeline organizes jobs into test, build, and update stages, and why a failed test stops the later stages.
  Why it matters: The stage order keeps an application that fails its tests from moving toward deployment.
• Image tag to chart handoff: Trace how the build job uses a shortened CI_COMMIT_SHA to tag and push the Docker image, then how the update job writes that tag into the Helm chart’s values.yaml and pushes the change to GitLab. Distinguish this CI handoff from the actual deployment performed by the CD tool.
  Why it matters: The chart change tells the deployment tool which newly built image to use.
• Repository credentials and variables: Describe the GitLab repository and CI/CD variable setup needed to push images and chart changes, including Docker Hub credentials and an API-scoped GitLab access token for pushing the updated chart.
  Why it matters: The automated jobs need credentials to publish the image and write the chart change back to GitLab.
• Pipeline triggers and checks: Describe how a code push triggers the GitLab CI pipeline and how to check it using an individual local GitLab Runner job or GitLab’s Run pipeline button and status UI.
  Why it matters: These checks let you verify that the automation runs and see whether its jobs succeed.

Key terms: .gitlab-ci.yaml, stages, test_code, build_image, update_helm_chart, pytest, CI_COMMIT_SHA, Docker Hub, Helm chart, values.yaml

3.4.2 Argo CD

This section shows how Argo CD deploys a CI-updated Helm chart and keeps Kubernetes resources aligned with Git.

• Git as deployment source: Explain how Argo CD compares the desired manifests in Git with resources running in Kubernetes and treats Git as the source of truth for deployment changes.
  Why it matters: This is the principle behind detecting and correcting deployment drift.
• Helm chart application setup: Describe how to connect the repository to Argo CD and create an application that deploys the Helm chart updated by CI.
  Why it matters: Argo CD needs both the chart’s location and a Kubernetes destination before it can deploy the application.
• Service drift and synchronization: Describe how changing the deployed FastAPI service from ClusterIP to LoadBalancer demonstrates drift, and distinguish detecting that drift from automatically syncing it back to Git.
  Why it matters: The example makes Argo CD’s comparison between live and expected manifests visible.

Key terms: Argo CD, source of truth, Git repository, manifests, sync, https, Applications, New App, Repository URL, target Revision

3.5 Prometheus and Grafana

This section shows how to expose application metrics, collect them with Prometheus, and visualize request activity in Grafana.

• Metrics alongside application logs: Distinguish logs used to debug application logic from metrics used to answer performance questions, such as request volume and response times, and explain why those metrics can support dashboards and alerts.
  Why it matters: Choosing metrics for measurable behavior makes production monitoring more useful than logs alone.
• Prometheus and Grafana roles: Explain how an application exposes metrics, Prometheus pulls and stores them as time series, Grafana queries Prometheus to build dashboards, and Alertmanager routes alerts.
  Why it matters: The roles of these tools explain how measurements become usable dashboards and notifications.
• FastAPI metrics instrumentation: Describe adding a metrics endpoint to the FastAPI hello-joker application with a prebuilt Starlette exporter, and distinguish counters, gauges, and histograms using the section’s examples. Describe how a custom counter can be defined and incremented with the Prometheus client library.
  Why it matters: Prometheus can collect useful application measurements only after the application exposes them.
• Cluster scraping and verification: Trace the section’s path from installing Prometheus and Grafana with Helm to deploying the instrumented application on Kubernetes, annotating its pod for Prometheus service discovery, and checking scraping through the /metrics endpoint and Prometheus Targets UI.
  Why it matters: An endpoint working locally is not enough; Prometheus must discover and successfully scrape the deployed application.
• Request count time series: Describe creating a Grafana dashboard panel backed by Prometheus and using a PromQL query with increase over a ten-minute range and sum by HTTP response code to plot recent request counts.
  Why it matters: The chart turns collected request counters into a readable view of recent traffic.

Key terms: logs, metrics, alerts, Prometheus, scraping, time series database, Grafana, Alertmanager, pull-based metrics system, /metrics

3.6 Summary

The chapter summary connects containerization, orchestration, CI/CD, and monitoring to reliable ML platforms while stressing that tools must serve organizational goals.