Chapter 16: Handling stateful applications with StatefulSets

Introduction

The introduction uses the Quiz service’s scaling difficulty to motivate the chapter’s focus on stateful workloads.

• Quiz service scaling constraint: Explain why the Quiz service has one replica while Kiada and Quote each have three, and state how all three services are currently deployed.
  Why it matters: The Quiz service shows why scaling a workload can be harder when its data must be considered.

Key terms: Deployment object, replicas, Quiz service, stateful workloads

16.1 Introducing StatefulSets

16.1.1 Understanding stateful workload requirements

Explains why stateful workloads are difficult to scale with Deployments, using a Quiz service backed by MongoDB to contrast shared storage with per-replica volumes and stable addresses.

• Stateful workload scaling requirements: Explain why a stateful workload must retain its state through restarts or relocation, and why adding or removing replicas requires attention to how that state is stored.
  Why it matters: The storage needs of each replica determine whether ordinary scaling will work.
• Shared volume scaling limits: Explain why scaling the Quiz Deployment to three pods does not give three working MongoDB instances sharing its existing volume.
  Why it matters: A higher replica count does not overcome the storage and application restrictions on sharing data.
• Per-replica storage and identity: Describe the storage and address each MongoDB replica needs, and explain why providing them with separate Deployments, Services, and PersistentVolumeClaims becomes cumbersome to scale.
  Why it matters: This workable but laborious arrangement motivates using a resource designed for stateful replicas.

Key terms: stateful workload, state, replicas, ReadWriteMany, ReadWriteOnce, PersistentVolumeClaim, PersistentVolume, data directory, MongoDB replica set, Service

16.1.2 Comparing StatefulSets with Deployments

This section contrasts Deployments and StatefulSets, explaining how StatefulSets preserve each replica’s identity and state, create pods in order by default, and handle storage when scaling.

• Pets and cattle distinction: Explain why Deployments suit interchangeable stateless pods, whereas StatefulSets suit stateful replicas that need their identity and state preserved when replaced.
  Why it matters: The distinction explains when replacing a pod with any equivalent pod is not enough.
• Per-replica identity and storage: Describe how a StatefulSet uses ordinal pod names and PersistentVolumeClaims to keep each replica associated with its own state across pod recreation.
  Why it matters: The pod-to-claim association is what lets a replacement resume with the state belonging to that particular replica.
• Default ordered pod creation: Describe the default order in which a StatefulSet creates its pods and what it waits for before creating the next one.
  Why it matters: A requested replica count does not mean every StatefulSet pod appears at once.
• Scaling and claim retention: Explain what a StatefulSet creates when scaled up and what can happen to its PersistentVolumeClaims when scaled down.
  Why it matters: Removing a replica does not, by itself, tell you whether its stored state is kept.

Key terms: Pets vs. Cattle, stateless workloads, stateful workloads, network identity, state, Pod template, PersistentVolumeClaim template, ordinal number, PersistentVolumeClaims, StatefulSet controller

16.1.3 Creating a StatefulSet

This section creates a headless Service and a quiz StatefulSet, explaining how the Service gives pods individual DNS identities and how the manifest configures pod creation and storage.

• Headless Service network identity: Explain why the quiz StatefulSet needs a headless Service and how that Service lets clients resolve individual pods by name. Describe how the StatefulSet associates with the Service.
  Why it matters: It explains how a stateful pod can be addressed individually rather than only through a shared Service name.
• DNS records before readiness: Explain how the quiz-pods Service is configured to include pods before they are ready.
  Why it matters: The setting determines whether a newly created quiz pod can be found through DNS before it becomes ready.
• Per-replica volume claim templates: Describe what volumeClaimTemplates creates and how its name relates to the Pod template.
  Why it matters: It connects each replica’s storage claim to the volume named in its pod configuration.
• Parallel pod creation policy: Distinguish the Parallel podManagementPolicy from the controller’s default pod-creation behavior and explain the choice in this example.
  Why it matters: Pod creation order can matter to distributed applications, even though this example uses the simpler parallel startup.

Key terms: headless Service, network identity, serviceName, clusterIP, publishNotReadyAddresses, quiz-pods, volumeClaimTemplates, PersistentVolumeClaims, Pod template, podManagementPolicy

16.1.4 Inspecting the StatefulSet, Pods, and PersistentVolumeClaims

Shows how to inspect a newly created StatefulSet, diagnose why its pods are not ready, and identify the pod and storage resources it creates.

• Pod readiness diagnosis: Trace why the quiz Pods initially fail to become ready and what changes after the MongoDB replica set is initiated.
  Why it matters: A failed readiness check does not necessarily mean the application cannot connect to its dependency.
• StatefulSet description and events: Describe what inspecting the StatefulSet with kubectl describe reveals about its storage template and controller activity.
  Why it matters: The description helps you verify both the storage template and the resources the controller has created.
• Pod identity and ownership: Explain how StatefulSet pod names, added labels, and owner references identify and relate its pods to the controller.
  Why it matters: These fields let you recognize an individual StatefulSet pod, its revision, and its controller.
• Individual pod storage claims: Explain how the StatefulSet turns its claim template into storage claims for individual pods.
  Why it matters: A pod’s volume reference points to its own claim, not to one claim shared by every replica.

Key terms: readiness probe, quiz-api, MongoDB replica set, kubectl rollout status, kubectl describe, PersistentVolumeClaim template, Event, ordinal index, controller-revision-hash, statefulset.kubernetes.io/pod-name

16.1.5 Understanding the role of the headless Service

Explains how a headless Service lets StatefulSet pods discover one another through DNS, then uses that discovery to import quiz data into a replicated MongoDB set.

• Individual StatefulSet Pod DNS: Explain why peer discovery matters for a distributed application and how a headless Service gives each StatefulSet pod a DNS name that resolves to that pod’s IP.
  Why it matters: Individual DNS records give cluster members a way to find specific peers without making the application depend on the Kubernetes API.
• MongoDB Discovery Through SRV Records: Describe how MongoDB uses the StatefulSet pods’ SRV records to discover replica addresses and ports, and why the Service port must be named mongodb.
  Why it matters: SRV discovery keeps the MongoDB connection string the same even when the replica count changes.
• One Time Quiz Data Import: Explain why quiz data is imported from a dedicated pod rather than each replica’s init container, and describe how the import and its success are checked.
  Why it matters: A separate importer avoids repeated imports while letting MongoDB replicate the imported data.

Key terms: peer discovery, headless Service, A or AAAA record, StatefulSet, SRV records, mongodb+srv, mongodb, connection string, quiz-data-importer Pod, mongoimport

16.2 Understanding StatefulSet behavior

This introduction previews tests of how the StatefulSet handles missing pods and node failures.

16.2.1 Understanding how a StatefulSet replaces missing pods

Explains how a StatefulSet replaces a deleted Pod while preserving its identity and storage, and how the volume type affects where the replacement can run.

• Replacement Pod identity and storage: Explain that a StatefulSet replaces a deleted Pod to maintain the desired replica count, giving the replacement the same name and PersistentVolumeClaim. Describe why clients using the Pod’s hostname can still reach it if its IP address changes.
  Why it matters: It explains how a deleted replica can return without losing its storage association or hostname-based reachability.
• Replacement Pod node placement: Distinguish where a replacement Pod can be scheduled when its claimed PersistentVolume is network-attached versus local to a node.
  Why it matters: The location of the storage determines whether a replacement Pod can move to another node.

Key terms: StatefulSet, PersistentVolumeClaim, replicas field, DNS records, PersistentVolume, network-attached volume, local volume

16.2.2 Understanding how a StatefulSet handles node failures

This section explains why a StatefulSet does not automatically replace a pod on an unreachable node and what can happen when an operator forces its replacement.

• At-most-one pod identity: Explain why a StatefulSet avoids automatically replacing a pod whose node has stopped reporting, and contrast this with a ReplicaSet.
  Why it matters: A node becoming unreachable is not proof that its stateful workload has stopped.
• Forced pod deletion safeguards: Describe when an operator must intervene to replace the unreachable pod and why ordinary deletion does not complete the job.
  Why it matters: Forcing deletion without establishing that the old workload has stopped can defeat the protection against duplicate instances.
• Volume constraints on recovery: Distinguish what can prevent the replacement pod from running with a local volume versus a network-attached volume, and describe the recovery options explained here.
  Why it matters: Replacing a Pod object alone does not guarantee that its existing storage is available to the new pod.

Key terms: at-most-one semantics, identity, StatefulSet controller, ReplicaSet, Terminating, Kubelet, Pod object, PersistentVolume, PersistentVolumeClaim, local volume

16.2.3 Scaling a StatefulSet

This section explains how StatefulSets scale pods and storage, and why scaling the pods alone may not make a stateful application ready.

• Ordinal Based Pod Scaling: Explain how to scale a StatefulSet and which pods are removed when it scales down.
  Why it matters: Pod ordinals make it predictable which replicas remain after scaling down.
• Persistent Claims Across Scaling: Describe what happens to PersistentVolumeClaims when a StatefulSet scales down and back up.
  Why it matters: Keeping claims lets returning replicas recover their existing data rather than start with new storage.
• Application Readiness After Scaling: Explain why changing the quiz StatefulSet's replica count does not by itself make MongoDB ready at the new size.
  Why it matters: Kubernetes can create the pods, but the application still has to accept the changed membership.
• Two Services For Different Needs: Distinguish why the quiz pods are exposed through both a regular Service and a headless Service.
  Why it matters: Separate Services let clients avoid unready pods without hiding those pods from their peers.

Key terms: StatefulSet, replicas, ordinal number, kubectl scale, PersistentVolumeClaim, PersistentVolume, persistentVolumeClaimRetentionPolicy, MongoDB replica set, quorum, readiness probe

16.2.4 Changing the PersistentVolumeClaim retention policy

Explains how a StatefulSet’s PersistentVolumeClaim retention policy controls claim deletion during scale-down and StatefulSet deletion, and why automatic deletion risks losing data.

• PersistentVolumeClaim retention settings: Explain how to configure a StatefulSet to retain or delete its PersistentVolumeClaims separately when it scales down and when it is deleted. Describe what happens in the quiz example.
  Why it matters: The two settings determine whether stored data remains available after different StatefulSet operations.
• Automatic deletion data risks: Justify the caution against setting either retention policy to Delete, including the risk of scaling to zero before deleting a StatefulSet. State when automatic deletion is appropriate and the alternatives given for retaining data.
  Why it matters: A seemingly safe deletion policy can still leave data exposed to loss through scale-down.
• Orphan deletion exception: Describe how deleting a StatefulSet with --cascade=orphan affects its pods and PersistentVolumeClaims.
  Why it matters: This deletion option preserves resources that an ordinary StatefulSet deletion might remove.

Key terms: persistentVolumeClaimRetentionPolicy, whenScaled, whenDeleted, Retain, Delete, reclaimPolicy, StorageClass, --cascade=orphan, PersistentVolumeClaims

16.2.5 Using the OrderedReady Pod management policy

Explains how OrderedReady sequences StatefulSet pods, why readiness can stall that sequencing, and where its ordering guarantees do and do not apply.

• OrderedReady pod sequencing: Explain how OrderedReady differs from Parallel when creating pods, and describe the order in which OrderedReady removes pods during scale-down.
  Why it matters: The policy determines whether replicas start and stop in a controlled sequence.
• Readiness deadlock and recovery: Trace why the quiz StatefulSet stalls with OrderedReady and describe the change and manual action used to get it moving again.
  Why it matters: A readiness check can prevent an ordered StatefulSet from starting the very replica it needs to become ready.
• Blocked OrderedReady scale-down: Describe what happens when an OrderedReady StatefulSet is scaled down while a replica is unready.
  Why it matters: An apparently idle scale-down may be waiting on readiness, even when the StatefulSet reports no reason.
• StatefulSet deletion ordering exception: Distinguish OrderedReady scale-down from deleting a StatefulSet when ordered pod termination is required.
  Why it matters: Deleting the StatefulSet directly does not provide the orderly shutdown that scaling it down does.

Key terms: podManagementPolicy, OrderedReady, Parallel, ordinal, minReadySeconds, readiness probe, quorum, Pod template, deadlock, scale-down

16.3 Updating a StatefulSet

Introduces declarative StatefulSet updates and distinguishes their update strategies from Deployment strategies.

• Declarative Pod template updates: Explain what happens when a StatefulSet’s Pod template is updated.
  Why it matters: This connects a change to the StatefulSet specification with the resulting change to its Pods.
• StatefulSet update strategy choices: Distinguish the StatefulSet update strategies introduced here from the Deployment strategies used for comparison.
  Why it matters: The strategy determines whether Pod replacement follows a rolling update or proceeds at your chosen pace.

Key terms: StatefulSet, Pod template, declarative updates, updateStrategy, RollingUpdate, OnDelete, Recreate

16.3.1 Using the RollingUpdate strategy

This section explains how a StatefulSet rolls out pod updates, what pauses a rollout, and how to inspect revisions or roll back.

• Sequential StatefulSet pod updates: Explain that a RollingUpdate replaces StatefulSet pods one at a time, starting with the highest ordinal, and waits for each replacement to be ready before proceeding.
  Why it matters: The replacement order and readiness gate explain why the service can remain accessible during the rollout.
• Readiness and rollout pauses: Describe what happens when existing pods are not ready before an update or a replacement pod never becomes ready, and how minReadySeconds can delay further replacement.
  Why it matters: It helps explain why a rollout may appear stuck rather than continuing to replace pods.
• Starting and tracking updates: Describe how the quiz StatefulSet update to version 0.2 is triggered and how its progress is checked.
  Why it matters: These commands connect the rollout behavior to an update a reader can initiate and observe.
• Revision history and rollback: Explain where StatefulSet revision history is stored and how to revert a StatefulSet to an earlier revision.
  Why it matters: Revision history makes it possible to inspect earlier versions and revert an unwanted update.

Key terms: RollingUpdate, ordinal number, ready, maxSurge, maxUnavailable, readiness, readiness probe, minReadySeconds, rollout, kubectl edit

16.3.2 RollingUpdate with partition

Explains how the RollingUpdate partition field lets you stage, test, and complete a StatefulSet update while preserving the correct template for each pod.

• Partition boundary for pod updates: Explain how the RollingUpdate partition value determines which StatefulSet pods are updated: pods with ordinals below it remain on the previous template, while pods at or above it can receive the new template.
  Why it matters: This boundary lets you predict exactly which pods a partitioned update will affect.
• Staged and canary rollout: Describe how to stage an update by setting partition to at least the replica count, deploy one canary by setting it to one less than the replica count, and complete the update by setting it to zero. Explain that changing the value can also advance an update in phases.
  Why it matters: You can test a new version on one pod before allowing it to reach the rest.
• Replacement pods across partitions: Explain which template the StatefulSet controller uses if a pod is deleted during a partitioned update.
  Why it matters: Replacing a pod does not bypass the rollout boundary you set.

Key terms: partition, ordinal number, RollingUpdate strategy, canary, replicas, Pod template, StatefulSet controller, previous template, new template

16.3.3 OnDelete strategy

This section explains how the OnDelete strategy gives you control over StatefulSet updates and rollbacks by replacing pods only when you delete them.

• OnDelete pod update process: Explain that applying a changed Pod template with the OnDelete strategy does not update existing pods; you choose which pods to delete and when, and the StatefulSet replaces each deleted pod using the new template.
  Why it matters: You can choose the timing and order of pod replacements to suit the workload.
• OnDelete rollback process: Describe why rolling back an OnDelete StatefulSet also requires deleting each pod that you want restored to the previous revision.
  Why it matters: An undo command alone will not finish replacing pods under OnDelete.
• Readiness during manual rollout: Explain that the OnDelete controller replaces a deleted pod regardless of readiness, so you must consider readiness before choosing to delete another pod.
  Why it matters: Manual control also makes you responsible for deciding whether it is safe to continue the rollout.

Key terms: OnDelete update strategy, Pod template, StatefulSet controller, ordinal number, kubectl rollout undo, OnDelete strategy, previous revision, pod readiness, OnDelete

16.4 Managing stateful applications with Kubernetes Operators

This section explains why StatefulSets alone may not fully manage a stateful application and introduces Kubernetes operators as a way to automate application-specific management.

• StatefulSet management limits: Explain why managing MongoDB can require work beyond a StatefulSet, using scaling and node failure as examples.
  Why it matters: These remaining tasks explain why a StatefulSet may not be enough for a fully automated stateful application.
• Operator-driven application management: Describe a Kubernetes operator and how a user deploys or configures an application through its custom object types.
  Why it matters: The custom object is the user-facing way to request application management while the operator handles the underlying Kubernetes workloads.

Key terms: StatefulSet, MongoDB replica set, quorum, Kubernetes operator, application-specific controller, custom object types, Kubernetes API, Deployments, StatefulSets

16.4.1 Deploying the MongoDB community operator

This section explains how to install the MongoDB community operator and prepare the cluster API for MongoDBCommunity objects, without yet deploying MongoDB.

• MongoDBCommunity API extension: Explain that creating a CustomResourceDefinition adds the MongoDBCommunity object kind to the cluster’s Kubernetes API, allowing MongoDBCommunity objects to be created later.
  Why it matters: It explains why the API must be extended before a MongoDBCommunity object can be created.
• Operator installation and verification: Describe the installation sequence: obtain the repository’s manifests, create the API extension and security-related objects, create the operator Deployment, and check its pod in the chosen namespace. Distinguish the running operator from a deployed MongoDB instance.
  Why it matters: A successful operator installation prepares the cluster to deploy MongoDB but is not itself a MongoDB deployment.

Key terms: CustomResourceDefinition, MongoDBCommunity, operator, object manifests, Deployment, mongodb namespace

16.4.2 Deploying MongoDB via the operator

This section shows how to deploy and manage a MongoDB replica set through a MongoDBCommunity custom object while the operator handles the underlying resources and MongoDB setup.

• MongoDBCommunity deployment configuration: Describe how to deploy MongoDB by creating a MongoDBCommunity object rather than creating the underlying StatefulSet and other objects yourself. Identify what its manifest configures and how it is applied.
  Why it matters: The custom object lets you describe the MongoDB deployment without assembling its underlying Kubernetes objects yourself.
• Operator reconciliation and ownership: Trace what happens after a MongoDBCommunity object is created, including which resources the operator creates, what the core controllers do, and why direct changes to operator-owned resources do not persist.
  Why it matters: This explains both where the MongoDB pods come from and why editing an operator-managed StatefulSet is ineffective.
• Replica set management through custom object: Explain how to change the size of the MongoDB replica set and what the operator does in response. State why no separate manual replica set initialization is needed.
  Why it matters: Changing the desired member count in one object updates both the Kubernetes workload and the MongoDB replica set.

Key terms: MongoDBCommunity, custom object, manifest, kubectl apply, reconciliation loop, operator, ownerReferences, StatefulSet, Services, Secrets

16.4.3 Cleaning up

This section gives the cleanup sequence for an operator-managed MongoDB deployment.

• MongoDB operator cleanup sequence: Describe the three cleanup steps: delete the MongoDBCommunity object so the garbage collector removes the underlying resources, delete the mongodb Namespace to remove the operator, and delete the CustomResourceDefinition to remove the custom object type from the API.
  Why it matters: Deleting MongoDB alone does not remove the operator or its custom object type.

Key terms: MongoDBCommunity, StatefulSet, Services, garbage collector, mongodb Namespace, CustomResourceDefinition

16.5 Summary

The section summarizes how StatefulSets provide stable identities and storage for stateful pods, control their lifecycle and updates, and fit into Operator-managed workloads.