StatefulSets

Introduction

StatefulSets is a practical Kubernetes capability, not just a definition to memorize. This lesson explains the problem it solves, why teams use it in production, how it behaves under failure, and how to practice it hands-on.

Purpose of this lesson

By the end, you should be able to decide when StatefulSets belongs in a design, implement it with a clear manifest or command flow, and troubleshoot the most common failure modes using Kubernetes status, events, logs, metrics, and ownership relationships.

Understanding the topic

StatefulSets run stateful workloads that need stable network identity, ordered rollout, and persistent storage per replica.

Use StatefulSets for workloads that need stable identity, ordered rollout, and one persistent volume per replica. Databases, brokers, and clustered systems often need this behavior.

• Read every object through metadata, spec, and status: who owns it, what should happen, and what the cluster reports actually happened.
• Connect YAML to the responsible component: scheduler, kubelet, controller manager, cloud controller, CSI driver, CNI, CoreDNS, admission webhook, or application runtime.
• Use it only when it improves a real operating concern such as availability, rollout safety, service discovery, isolation, security, cost, or developer workflow.

Visual explanation

Use this mental model when explaining the lesson during design reviews or incidents:

StatefulSet postgres
  -> postgres-0 + pvc/data-postgres-0
  -> postgres-1 + pvc/data-postgres-1
  -> headless Service gives stable DNS

Step-by-step explanation

1. Identify the workload or platform problem first: availability, traffic routing, storage, configuration, identity, policy, scaling, observability, or troubleshooting.
2. Write the smallest useful desired state for StatefulSets; include labels, namespace, ownership, resource settings, and health checks where relevant.
3. Apply or render the change in a safe environment, then read status and events before assuming the manifest worked.
4. Break one realistic dependency such as a selector, image tag, probe, permission, quota, or endpoint and practice the recovery path.
5. Promote through Git or your release process with a rollback plan, alert coverage, and a short runbook.

Informative example

StatefulSet for stable pod identities: After applying it, verify both desired and observed state. A production-ready workflow should include kubectl diff, apply, describe, get events, and a rollout or health check when the object supports it.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  labels:
    app: postgres
    app.kubernetes.io/name: postgres
spec:
  serviceName: postgres
  revisionHistoryLimit: 3
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: app
          image: nginx:1.27-alpine
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 20
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi

Real-world use

A Kafka broker set needs predictable pod names, stable DNS, and durable volumes so broker identity survives restarts and rescheduling.

Best practices

• Keep manifests reviewed in Git and treat manual cluster changes as temporary break-glass actions.
• Use standard labels such as app.kubernetes.io/name, part-of, and managed-by so selectors, dashboards, alerts, and cost reports line up.
• Attach ownership, environment, and runbook metadata before resources reach production.

Common mistakes

• Confusing resource exists with resource is healthy. Always inspect status, events, and downstream dependencies.
• Changing selectors, labels, or names casually; these are contracts between controllers, Services, policies, dashboards, and GitOps tools.
• Debugging from memory instead of reading the object: kubectl describe, events, endpoints, and controller logs usually tell the story.

Debugging tips

• Start with kubectl describe and recent events sorted by time; they often identify scheduling, image, probe, volume, or policy failures.
• Compare desired state with live state using kubectl get -o yaml, kubectl diff, and the owning controller's status.
• Follow the traffic or lifecycle path one hop at a time rather than jumping straight to the node or the application code.

Optimization strategies

• Tune requests, limits, probes, and rollout settings from observed production behavior instead of copying defaults.
• Reduce blast radius with namespaces, quotas, PodDisruptionBudgets, topology spread, and progressive delivery.
• Automate validation with CI, policy checks, and GitOps drift detection so correctness is enforced before outages.

Advanced interview questions

Hands-on exercise

Scale a StatefulSet from one to three replicas and observe ordered creation, pod names, PVC names, and DNS records.

# Suggested lab loop for StatefulSets
kubectl create namespace statefulsets-lab
kubectl -n statefulsets-lab apply -f lesson.yaml
kubectl -n statefulsets-lab get all
kubectl -n statefulsets-lab describe all
kubectl -n statefulsets-lab get events --sort-by=.lastTimestamp

Summary

StatefulSets becomes valuable when you connect the API object or command to real operational behavior. Practice the happy path, then deliberately break it so troubleshooting becomes evidence-driven rather than guesswork.