Chapter 17: Deploying per-node workloads with DaemonSets

Introduction

Introduces DaemonSets as the Kubernetes resource for running one agent or daemon Pod on each cluster node.

• Per-node DaemonSet workloads: Explain when to use a DaemonSet: to run one replica on each node, such as an agent that collects metrics or aggregates logs for that node.
  Why it matters: This identifies the per-node workload that DaemonSets are intended to run.

Key terms: DaemonSet, agent, daemon, replica, node

17.1 Introducing DaemonSets

This section introduces DaemonSets as the way to run one pod replica per cluster node, with an option to limit which nodes receive one.

• DaemonSet per-node placement: Explain that a DaemonSet is a Kubernetes API object that ensures exactly one pod replica runs on each cluster node by default, and that a node selector can restrict deployment to some nodes.
  Why it matters: This distinguishes a per-node workload from simply choosing a total number of pod replicas.

Key terms: DaemonSet, daemon Pods, node selector

17.1.1 Understanding the DaemonSet object

This section explains how a DaemonSet maintains one pod per node and why that pattern suits node-level infrastructure services.

• One pod per node: Distinguish a DaemonSet from replica-counted workloads: it uses a Pod template to create as many pods as there are cluster nodes and places each on a different node.
  Why it matters: The placement goal determines when a DaemonSet is the right workload object.
• Node infrastructure workloads: Explain why DaemonSets are typically used for system-level services needed on each node, and why managing those services as DaemonSets can be preferable to running them through host init scripts or systemd.
  Why it matters: It connects the one-pod-per-node pattern to practical cluster services.
• DaemonSet reconciliation behavior: Trace how the DaemonSet controller uses a label selector and Pod template to maintain exactly one matching pod per node, including its responses to node and pod changes.
  Why it matters: This explains how per-node coverage is maintained as the cluster changes.

Key terms: DaemonSet, Pod template, replicas, nodes, ReplicaSet, infrastructure pods, system-level service, Kube Proxy, Container Network Interface (CNI) plugin, DaemonSet controller

17.1.2 Deploying pods with a DaemonSet

This section creates and inspects a DaemonSet, then explains how its status, node eligibility, and pod manifests reveal where daemon pods run.

• DaemonSet manifest and selector: Describe the DaemonSet manifest’s label selector and Pod template, including why the template’s labels must match the selector and what changing the selector requires.
  Why it matters: Matching labels let the DaemonSet associate its pods with the object that manages them.
• Node-based DaemonSet status: Explain why DaemonSet status fields count nodes served rather than simply counting pods, including what can happen during a Pod template update.
  Why it matters: Node-based counts prevent you from mistaking temporary extra pods during an update for extra nodes being served.
• Control plane scheduling restriction: Explain why the example DaemonSet can create fewer pods than the cluster has nodes even without a node selector.
  Why it matters: It explains why having no node selector does not guarantee a daemon pod on every cluster node.
• Taints and pod tolerations: Describe, at the level shown here, how a DaemonSet can permit its pods to run on otherwise restricted nodes, and distinguish this from selecting nodes.
  Why it matters: Tolerations explain how a critical per-node service can reach nodes that reject ordinary pods.
• Daemon pod ownership and targeting: Describe what an inspected daemon pod reveals about its ownership and placement, including why the controller adds node affinity rather than scheduling it directly.
  Why it matters: A pod manifest shows both which DaemonSet owns the pod and how that pod is directed to its node.

Key terms: DaemonSet, label selector, Pod template, status, nodes, control plane node, worker nodes, node selector, taints, tolerations

17.1.3 Deploying to a subset of Nodes with a node selector

This section shows how a DaemonSet uses node labels to run pods on selected nodes and how changes to labels or the node selector change which nodes are targeted.

• DaemonSet node eligibility: Explain how a node selector limits a DaemonSet to matching nodes and distinguish it from the pod selector.
  Why it matters: This distinction lets you target nodes without confusing node eligibility with pod ownership.
• Node label change response: Trace what happens when a node gains or loses a label required by a DaemonSet’s node selector.
  Why it matters: Changing a node’s labels can move it into or out of a DaemonSet’s scope.
• Standard labels for targeting: Describe how standard node labels can target DaemonSet pods by OS or architecture, and when separate DaemonSets make sense instead of one multiarch image.
  Why it matters: Platform labels help place node agents on compatible machines without unnecessarily duplicating DaemonSets.
• Mutable node selector updates: Explain that a DaemonSet’s node selector can be changed, including the section’s example of removing it with a JSON patch.
  Why it matters: You can change which nodes a DaemonSet targets without replacing the DaemonSet.

Key terms: node selector, pod selector, Pod template, spec.nodeSelector, Node objects, reconciliation loop, kubernetes.io/arch, kubernetes.io/os, multiarch container images, pod label selector

17.1.4 Updating a DaemonSet

This section explains how RollingUpdate and OnDelete replace DaemonSet pods, and why manual updates can be safer for critical daemons.

• Rolling update pod overlap: Explain how maxSurge controls whether an old daemon Pod is stopped before its replacement starts or overlaps with it.
  Why it matters: A rollout that overlaps daemon instances may fail even when both pod versions can start individually.
• Rolling update node availability: Describe how maxUnavailable limits the pace of a DaemonSet rollout and how increasing it changes the number of nodes updated at once.
  Why it matters: The setting trades rollout speed against how many nodes a faulty daemon update can affect at once.
• Unready pod replacement exception: Explain what happens during a rolling update if an existing daemon Pod’s readiness probe fails.
  Why it matters: An unready existing Pod can be replaced outside the rollout pace you configured.
• Manual OnDelete pod replacement: Describe what a DaemonSet controller does after an OnDelete template update and how the operator causes pods to receive the new template.
  Why it matters: OnDelete lets an operator choose the order and pace of replacement instead of relying on an automatic rollout.
• Critical daemon update control: Justify why OnDelete may be preferable for cluster-critical daemon Pods despite requiring more effort.
  Why it matters: Checking the rest of a node before proceeding can prevent a bad daemon update from cascading across the cluster.

Key terms: RollingUpdate, maxSurge, minReadySeconds, maxUnavailable, ready and available, Recreate, readiness probe, OnDelete, Pod template, cluster-critical pods

17.1.5 Deleting the DaemonSet

This subsection closes the DaemonSet demo by deleting the DaemonSet and checking that its pods are gone.

• DaemonSet deletion and pod removal: Explain that deleting the demo DaemonSet also deletes its demo pods, and describe how to confirm their removal.
  Why it matters: Checking the pod list verifies that cleanup removed the demo workload.

Key terms: DaemonSet, Pods

17.2 Special features in pods running node agents and daemons

This section explains why node agents and daemons may need greater access to their host node and where that access must be requested.

• Node access for daemon pods: Explain why node agents and daemons may need more access to their node than regular workloads, and state that the required exceptions must be specified in the pod manifest.
  Why it matters: This explains why a node daemon’s pod may need access settings that an ordinary workload does not.

Key terms: node agents, daemons, Linux namespaces, pod manifest

17.2.1 Giving containers access to the OS kernel

This section explains why containers need explicit permission for sensitive kernel operations and contrasts privileged containers with granting specific capabilities.

• Privileged kernel access: Explain why containers cannot perform some kernel operations by default and how a privileged container gains full kernel access. Describe why the kube-proxy container uses that access.
  Why it matters: It explains both the need for kernel-access restrictions and why kube-proxy is an exception.
• Selective kernel capabilities: Justify granting a node agent only the capabilities it needs instead of making it privileged, using kindnet's network capabilities as an example.
  Why it matters: Granting only needed capabilities avoids giving a node workload unnecessary kernel access.

Key terms: system calls, securityContext, privileged, kube-proxy, capabilities, NET_RAW, NET_ADMIN, kindnet

17.2.2 Accessing the node’s filesystem

Explains why a daemon Pod may need access to the node’s filesystem and illustrates that access with kube-proxy’s hostPath volumes.

• Node filesystem access for daemons: Explain why a node agent or daemon might need access to the host filesystem and describe what kube-proxy accesses through its two hostPath volumes.
  Why it matters: Some per-node daemons need files from the node itself to do their work.

Key terms: hostPath volume, daemon Pod, kube-proxy, xtables.lock, kernel modules

17.2.3 Using the node’s network and other namespaces

Explains how a DaemonSet pod can use its node’s network while remaining isolated by other namespaces unless explicitly configured otherwise.

• Host network for daemon pods: Explain what setting hostNetwork to true does and why a daemon Pod might use it.
  Why it matters: It explains why a daemon Pod can be reached through its host node’s address.
• Other namespace isolation: Describe which isolation remains when a pod uses host networking and how a daemon Pod can instead use the node’s IPC and PID namespaces.
  Why it matters: Sharing the host network does not by itself give a daemon access to the node’s processes.

Key terms: hostNetwork, node’s network environment, node’s address(es), hostIPC, hostPID, IPC namespaces, PID namespaces

17.2.4 Marking daemon Pods as critical

Explains why critical node daemons need higher pod priority and how to assign it with PriorityClasses.

• Priority for system Pods: Explain why a critical system Pod may need higher priority than a regular workload Pod, and why running it through a DaemonSet does not provide that priority by default.
  Why it matters: Running a node agent everywhere does not, by itself, ensure it can get space on a full node.
• PriorityClass selection for node agents: Describe how a PriorityClass affects a Pod’s priority and how a daemon Pod is assigned one. Distinguish the two built-in system classes and explain why kube-proxy uses system-node-critical.
  Why it matters: Choosing an appropriate class helps keep a node-critical agent running when it competes with other Pods for space.

Key terms: system pods, DaemonSet, Pod priority classes, PriorityClass, priorityClassName, system-cluster-critical, system-node-critical, preemption policy, kube-proxy

17.3 Communicating with the local daemon Pod

Introduces why client pods need a way to reach a daemon Pod on their own node and the node-agent example used to explore that problem.

• Local daemon connection requirement: Explain why a workload that needs a node-local daemon cannot rely on ordinary Service routing to reach it.
  Why it matters: This is the problem the section’s communication approaches need to solve.
• Node agent demonstration: Describe what the example node agent supplies and how the Kiada application uses it.
  Why it matters: The example makes the need for communication with a node agent concrete.

Key terms: daemon Pod, Service, client pod, node, node agent, Kiada, uptime, average node utilization

17.3.1 Binding directly to a host port

This subsection shows how to expose each node-agent Pod through its node’s host port and configure Kiada to contact the agent on its own node.

• Local host-port forwarding: Explain how hostPort exposes a node-agent container through a port on its host node, and distinguish this local forwarding from a NodePort Service.
  Why it matters: The host port gives a client a way to reach the agent on its own node rather than one on another node.
• Node-specific agent URL: Explain how Kiada constructs the URL for its local agent without fixing a node IP address in the Pod manifest.
  Why it matters: Using the Pod’s actual node IP lets each Kiada instance contact its own node’s agent.
• Local-agent connectivity checks: Describe how the example checks that the node agent is reachable and that Kiada receives information from an agent on the same node.
  Why it matters: These checks distinguish an agent that merely runs from one that Kiada can reach locally.

Key terms: hostPort, host port, NodePort Service, NODE_AGENT_URL, NODE_IP, Downward API, DaemonSet, GET /, Node info, Request processed by

17.3.2 Using the node’s network stack

This subsection shows how a node-agent pod can use its host node’s network environment to serve clients through the node’s IP address and its listening port.

• Host network daemon access: Explain how using the host node’s network environment makes the agent reachable, and distinguish it from exposing the agent with hostPort.
  Why it matters: The client connects the same way in both setups, but the agent listens on a different port.

Key terms: host node’s network environment, node’s IP address, hostPort, --listen-address

17.3.3 Using a local Service

Shows how to expose DaemonSet agents through a node-local Service, configure a client to use it, and choose it over host networking when external access is unnecessary.

• Node-local Service routing: Explain how a Service with internalTrafficPolicy set to Local connects a client only to an agent Pod on the same node, and what happens when that node has no agent.
  Why it matters: The local policy lets clients use a Service without accidentally reaching an agent on another node.
• Agent Service configuration: Describe how the example moves from a node-bound agent port to a Service: the agent binds to port 80 of its Pod IP, the Service selects agent Pods by label with Local traffic policy, and Kiada uses the Service name in NODE_AGENT_URL.
  Why it matters: These changes preserve Kiada’s access to its local agent after removing the node-bound port.
• Local Service tradeoffs: Justify choosing a local Service over hostPort or hostNetwork when external access is unnecessary, and describe the security concern with hostNetwork.
  Why it matters: The choice affects both how exposed the agent is and what access its Pod has to the node’s network.

Key terms: Service, internalTrafficPolicy, Local, node-agent, node selector, hostPort, hostNetwork, app: node-agent, NODE_AGENT_URL, local Service

17.4 Summary

The section summarizes how DaemonSets place and update per-node Pods, how those Pods access node resources, and how clients reach a local daemon.