Kubernetes Patterns

Introduction

Kubernetes patterns are the operational primitives staff architects use to run distributed systems on container orchestration — deployments, services, ingress, HPA, PDB, and workload identity. Google created Kubernetes from Borg's lessons; production patterns differ sharply from tutorial manifests.

This lesson teaches pod lifecycle patterns, rollout strategies, resource governance, and how Google-scale teams avoid the "YAML spaghetti" anti-pattern via platform abstractions and admission policies.

Real production story

A Google Cloud customer running GKE migrated 200 microservices from EC2 with copy-pasted Deployment YAML per team. During a cluster upgrade, 40 services had no PodDisruptionBudget — node drains evicted all replicas simultaneously, causing a 12-minute checkout outage. Google's SRE review introduced paved-road Helm charts with mandatory PDB, topology spread constraints, and readiness probes that actually reflect dependency health — not just "port 8080 open".

Business problem

Business pressure: Google Cloud and internal Borg/GKE users must run thousands of workloads with safe rollouts, autoscaling, and cluster upgrades without per-team SRE expertise. Ad-hoc Kubernetes YAML creates correlated failures during infrastructure maintenance.

• Revenue at risk: Bad rollout or missing PDB causes multi-service outage during routine node maintenance.
• Engineering velocity: Each team reinventing deployment patterns slows migration to Kubernetes and increases incident rate.
• Compliance / trust: Workload identity, network policies, and resource quotas must be enforced platform-wide — not optional.

Architecture overview

Kubernetes patterns in production include: rolling updates with maxUnavailable/maxSurge, readiness/liveness/startup probes, PodDisruptionBudgets, topology spread, resource requests/limits, HPA/VPA, and workload identity for cloud API access.

• Definition: Repeatable K8s resource compositions that encode safe deploy, scale, and recovery semantics.
• When to adopt: Running 10+ services on shared clusters with platform SRE ownership.
• When to defer: Single service, single node — managed PaaS may be simpler.
• Operability: Rollout success rate, eviction events, HPA scaling lag, and probe failure rate per deployment.

Architecture motivation

Why architects care: Kubernetes abstracts compute but exposes sharp edges — probes, limits, affinity, and rollout semantics determine whether a deploy is safe. The naive "Deployment + Service" pattern ignores disruption budgets, topology spread, and graceful termination.

• Force: Hundreds of services on shared clusters with frequent node upgrades and autoscaling events.
• Constraint: Cannot hire a K8s expert per product team — need paved-road templates and policy enforcement.
• Outcome: Golden-path charts, OPA/Gatekeeper policies, and SLO-based HPA defaults.

Internal architecture

Google GKE production deployment pattern — safe rollout with governance:

• maxUnavailable: 0 ensures no capacity drop during rollout — surge handles replacement.
• Readiness checks dependencies — liveness only checks process health.
• PDB minAvailable prevents node drain from taking all replicas.

Ingress (Gateway API) → Service (ClusterIP)
   ↓
Deployment (replicas: 3)
  strategy: RollingUpdate
    maxUnavailable: 0
    maxSurge: 1
  podTemplate:
    topologySpreadConstraints: [zone, hostname]
    containers:
      resources: { requests, limits }
      readinessProbe: HTTP /ready (checks DB + cache)
      livenessProbe: HTTP /healthz
      startupProbe: HTTP /startup (slow JVM)
    terminationGracePeriodSeconds: 60
    preStop: sleep 15 (drain in-flight)

PodDisruptionBudget: minAvailable: 2
HorizontalPodAutoscaler: CPU 70% + custom metric (queue depth)
ServiceAccount + Workload Identity → GCP IAM

Data flow

Deploy: CI pushes image digest → GitOps/helm upgrade → rolling update creates new pods → readiness passes → old pods terminated. Scale: HPA watches metrics → adjusts replicas. Drain: node cordon → PDB respects minAvailable → pods rescheduled.

• Write path: Argo CD syncs manifest from git SHA → Deployment rollout with revision history.
• Read path: Service endpoints update only when readiness probe succeeds.
• Async path: preStop hook + grace period drain connections before SIGTERM.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }
  template:
    spec:
      serviceAccountName: checkout-api
      terminationGracePeriodSeconds: 60
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector: { matchLabels: { app: checkout-api } }
      containers:
        - name: app
          image: gcr.io/project/checkout@sha256:abc...
          resources:
            requests: { cpu: "500m", memory: "512Mi" }
            limits: { cpu: "2", memory: "1Gi" }
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
          lifecycle:
            preStop:
              exec: { command: ["sleep", "15"] }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api-pdb
spec:
  minAvailable: 2
  selector: { matchLabels: { app: checkout-api } }

System design diagram

Two diagrams show the Kubernetes Patterns topology and the primary request/event path used in production at scale.

Production code example

OPA Gatekeeper policy — enforce production K8s patterns:

• Admission policies catch missing probes and PDB at apply time — not during node drain.
• Pair with golden-path Helm chart so compliant YAML is the easy path.
• Exception process documented — not silent policy bypass.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredProbes
metadata:
  name: must-have-probes
spec:
  match:
    kinds: [{ apiGroups: ["apps"], kinds: ["Deployment"] }]
  parameters:
    probes: ["readinessProbe", "livenessProbe"]
    probeTypes: ["httpGet", "tcpSocket"]

# Reject deployments without PDB in production namespace
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: production-pdb-required
spec:
  match:
    namespaces: ["production"]
  parameters:
    labels: ["pdb.minAvailable"]

Enterprise case study

Google — GKE paved-road deployment templates: Platform team ships Helm chart with PDB, probes, topology spread, and HPA pre-configured. OPA Gatekeeper rejects deployments missing required fields.

• Before: 40% of services had no PDB; cluster upgrades caused correlated outages.
• Decision: Golden-path chart + admission policy; exceptions require architecture review.
• After: Zero PDB-related outages in 8 cluster upgrades; rollout rollback time < 2 minutes.

Trade-offs

• Rolling vs blue-green: Rolling is simpler; blue-green needs dual deployments or service mesh traffic split.
• HPA on CPU vs custom metrics: CPU lags for IO-bound; queue depth or RPS metrics react faster.
• Requests vs limits: Requests schedule; limits cap — omitting requests causes noisy neighbor scheduling.
• GitOps vs imperative: GitOps gives audit trail; imperative kubectl is fine for emergencies only.

Security considerations

Kubernetes is the control plane: RBAC, network policies, pod security standards, and workload identity are architectural requirements.

• Identity: Workload Identity / IRSA — no cloud credentials in pod env vars.
• Data: NetworkPolicy default-deny with explicit egress to known services.
• Supply chain: Admission controller blocks :latest tags and unsigned images from unapproved registries.

Scalability analysis

Scale dimensions: Google GKE clusters run 5000+ pods. API server load, etcd size, and DNS QPS become bottlenecks before individual pod CPU.

• Horizontal scale: HPA per deployment; cluster autoscaler adds nodes when pending pods cannot schedule.
• Hot spots: Single-zone deployment concentrates load — topology spread across zones mandatory.
• Cost: Right-size requests — over-provisioned requests block scheduling and waste node capacity.

Failure scenarios

What breaks: Liveness kills slow-starting pods; missing PDB during node upgrade; image pull failures during rollout with maxUnavailable too high.

• Probe misconfiguration: Liveness hits /ready with DB check — restart loop during DB blip — separate endpoints.
• PDB missing: Node drain evicts all replicas — set minAvailable or maxUnavailable on every production deployment.
• Rollout stuck: New pods never pass readiness — check rollout status; rollback to previous revision.

Staff engineer insights

• Readiness probe must check dependencies — "port open" is not readiness.
• PDB on every production Deployment — no exceptions without signed risk acceptance.
• maxUnavailable: 0 + maxSurge: 1 is the safe default for stateless HTTP services.
• Platform golden-path charts beat 200 copies of almost-correct YAML.

Interview questions

Architecture review questions

• PDB defined with minAvailable aligned to replica count?
• Readiness probe checks real dependencies, not just port open?
• Resource requests set (not just limits)?
• Topology spread across zones for HA?
• Workload identity — no static cloud credentials in pods?
• Rollout strategy maxUnavailable: 0 for production HTTP services?

Summary

Kubernetes patterns at Google scale mean PDB-backed rollouts, dependency-aware readiness probes, topology spread, and admission-enforced golden paths. The orchestrator is only as safe as the patterns your platform mandates — not the tutorials your teams copied.