Kubernetes

Introduction

Kubernetes (K8s) orchestrates Docker containers across a cluster — scheduling Java services as Pods, managing releases with Deployments, exposing traffic via Services, and injecting config through ConfigMaps and Secrets.

Spring Boot on Kubernetes uses actuator health probes, graceful shutdown on SIGTERM, and service discovery via DNS (payment-service.default.svc.cluster.local). This lesson maps K8s primitives to how Java microservices actually run in production — not abstract YAML trivia.

Business problem

Manual container management doesn't scale:

• Node failure: Single VM dies — payment service down until manual restart.
• Rolling deploy: Zero-downtime deploy of 20 Java pods — manual script error causes outage.
• Config sprawl: JDBC URL baked in image — rebuild to change database endpoint.
• Secret leakage: DB password in git — compliance failure.
• Resource hog: One Spring Boot pod OOMs node — no isolation between services.

Why this topic exists

Kubernetes automates container lifecycle at fleet scale:

• Pod: Smallest deploy unit — one or more containers sharing network/IP (usually one Java app per pod).
• Deployment: Declarative desired state — 3 replicas, rolling update strategy, rollback history.
• Service: Stable ClusterIP/DNS load balances to healthy pod endpoints.
• ConfigMap: Non-sensitive config — application.yml overrides, feature flags.
• Secret: Base64-encoded sensitive data — mount as env or files for JDBC password.

Core concepts

K8s concepts for Java engineers:

• Deployment → ReplicaSet → Pods: Deployment manages ReplicaSet manages Pods — scale with replicas: N.
• Service types: ClusterIP (internal), NodePort, LoadBalancer, Ingress (HTTP routing).
• Probes: liveness (restart if dead), readiness (remove from Service if not ready), startup (slow Spring boot).
• ConfigMap mount: Volume mount /config/application.yml — Spring spring.config.additional-location.
• Secret as env: SPRING_DATASOURCE_PASSWORD from secretKeyRef — never in image.
• Resource requests/limits: requests for scheduling; limits for cgroup cap — align with JVM MaxRAMPercentage.

Internal architecture

Java payment service on Kubernetes:

Internet ──▶ Ingress ──▶ Service (payment-svc:80)
                              │
                    ClusterIP load balance
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
         Pod (payment)  Pod (payment)  Pod (payment)
         Spring Boot    Spring Boot    Spring Boot
         :8080          :8080          :8080
              │               │               │
              └───────────────┼───────────────┘
                              ▼
                    ConfigMap: app-config
                    Secret: db-credentials
                              │
                              ▼
                    PostgreSQL (StatefulSet or managed RDS)

Deployment spec highlights:
  replicas: 3
  strategy: RollingUpdate (maxSurge 1, maxUnavailable 0)
  template.spec.containers[0]:
    resources.requests.memory: 512Mi
    resources.limits.memory: 512Mi
    livenessProbe: /actuator/health/liveness
    readinessProbe: /actuator/health/readiness

Pods, Deployments, Services, and config injection:

Code walkthrough

Deployment + Service + ConfigMap + Secret for Spring Boot:

• image digest pin: Immutable deploy — same artifact verified in staging.
• secretKeyRef: Password injected at runtime — not in ConfigMap plaintext.
• liveness vs readiness: Liveness restart pod; readiness remove from Service endpoints during slow startup.
• preStop sleep: Allow load balancer to deregister before SIGTERM — graceful drain.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
    spec:
      containers:
        - name: payment
          image: registry.example.com/payment:1.2.3@sha256:abc...
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: payment-config
          env:
            - name: SPRING_DATASOURCE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-credentials
                  key: password
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "512Mi"
              cpu: "1000m"
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 60
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 30
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 5"]  # drain in-flight requests

---
apiVersion: v1
kind: Service
metadata:
  name: payment-service
spec:
  selector:
    app: payment-service
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP

Production example

ConfigMap + Spring Boot externalized config:

• graceful shutdown: server.shutdown=graceful + terminationGracePeriodSeconds ≥ 30 — in-flight payments complete.
• application.yaml in ConfigMap: Mount as volume or use spring.cloud.kubernetes for hot reload.
• HPA: Scale replicas on CPU — each pod needs headroom for JVM GC spikes.
• Sealed Secrets: Encrypt Secret for git storage — decrypt only in cluster.

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: payment-config
data:
  SPRING_PROFILES_ACTIVE: "production"
  SPRING_KAFKA_BOOTSTRAP_SERVERS: "kafka:9092"
  LOGGING_LEVEL_ROOT: "INFO"
  application.yaml: |
    server:
      shutdown: graceful
    spring:
      lifecycle:
        timeout-per-shutdown-phase: 30s

---
# secret.yaml (apply with sealed-secrets or external-secrets in prod)
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:
  password: "REPLACE_VIA_CI_NOT_GIT"

# Spring Boot 3 actuator probes — application.properties
management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true

# HPA — scale Java pods on CPU
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Enterprise case study

Uber — Kubernetes at scale: Uber runs thousands of microservices on Kubernetes — Java and Go services share platform primitives. Key lesson for Java teams: readiness probe must reflect real readiness — Spring actuator readiness includes DB/Kafka health; premature ready → traffic to starting pod → 503 storm. Set initialDelaySeconds and startupProbe for slow Spring context.

• Before: Mesos custom scheduler — inconsistent deploy patterns per team.
• After: Standard Deployment + HPA + platform ConfigMaps — paved road for Java Spring Boot.
• Java-specific: memory limit = heap (MaxRAMPercentage) + 128MB buffer minimum.
• Incident: Missing preStop — rolling deploy dropped in-flight requests during SIGTERM.

Performance considerations

K8s performance for Java workloads:

• Right-size requests: Over-requested memory strands cluster capacity; under-requested → eviction.
• CPU requests affect GC: JVM sees AvailableProcessors from CPU limit — too low slows GC.
• Pod density: Don't pack heap-heavy Java pods — leave node headroom for OS and kubelet.
• Anti-affinity: Spread payment pods across nodes — single node failure doesn't take all replicas.
• Startup probe: Long initialDelay on liveness causes slow start kill — use startupProbe instead.

Security considerations

K8s security for Java services:

• RBAC: Service account per app — minimal permissions; no cluster-admin for apps.
• NetworkPolicy: Payment pod talks only to postgres and kafka — deny all else.
• Secret management: External Secrets Operator, Vault — not plaintext in git.
• Pod security: runAsNonRoot, readOnlyRootFilesystem, drop ALL capabilities.
• Image pull secrets: Private registry auth for Java app images.

Scalability considerations

Scaling Java on Kubernetes:

• HPA on CPU/custom metrics: Kafka consumer lag metric for worker deployments.
• VPA (Vertical): Recommend memory requests after observing JVM heap usage — caution with Java.
• Cluster autoscaler: Add nodes when pods pending — Java services memory-heavy.
• Multi-zone: podAntiAffinity + topologySpreadConstraints — AZ failure tolerance.

Production challenges

Real K8s + Java challenges:

• CrashLoopBackOff: Spring fails startup — check logs, missing Secret, wrong ConfigMap key.
• Probe misconfiguration: Liveness kills during GC pause — tune or use startupProbe.
• Config drift: ConfigMap updated but pods not restarted — stale config until rollout.
• Memory limit too tight: OOMKilled during peak — monitor JVM heap vs container limit.
• DNS resolution delay: Service name fails on first boot — retry logic in Java client or init container.

Common mistakes

• Same value for liveness and readiness with short timeout — restart loop on slow DB connect.
• Secrets in ConfigMap — base64 is not encryption; use Secret resource properly.
• No resource limits — noisy neighbor Java pod consumes entire node.
• Hardcoding localhost in Spring — use K8s Service DNS names.
• Ignoring terminationGracePeriodSeconds — Spring graceful shutdown cut mid-request.

Debugging guide

Debug Java pods on Kubernetes:

• Pod logs: kubectl logs payment-service-xxx -f --previous for crash.
• Describe pod: Events show OOMKilled, FailedMount, probe failures.
• Exec debug: kubectl exec -it pod -- wget -qO- localhost:8080/actuator/health.
• Port forward: kubectl port-forward svc/payment-service 8080:80 local test.

# Pod status and events
kubectl describe pod -l app=payment-service

# Check endpoints registered
kubectl get endpoints payment-service

# ConfigMap mounted correctly?
kubectl exec payment-pod -- env | grep SPRING

# Rollout status
kubectl rollout status deployment/payment-service
kubectl rollout undo deployment/payment-service  # rollback

Best practices

• One Java application per Pod — sidecar for mesh/logging only when needed.
• Deployment with rolling update maxUnavailable=0 for payment services.
• Service ClusterIP for internal Java-to-Java — Ingress for external HTTP.
• Externalize config via ConfigMap; credentials via Secret.
• Enable Spring Boot actuator probes — separate liveness and readiness.
• Set memory requests = limits for Java (Guaranteed QoS) — predictable JVM.
• preStop hook + graceful shutdown — drain before SIGTERM.

Anti-patterns

• Deploying without probes — broken pods receive traffic.
• Latest image tag in Deployment — use digest or immutable semver.
• Giant ConfigMap with secrets mixed in — split ConfigMap and Secret.
• Single replica production Java service — no HA during node drain.
• Manual kubectl edit prod Deployment — GitOps (ArgoCD/Flux) for audit trail.

Staff engineer notes

• Staff engineers define "paved road" Deployment template for all Java services — probes, resources, labels standard.
• Memory limit for Java pod is a JVM tuning parameter — coordinate with MaxRAMPercentage, don't copy from Node.js template.
• Readiness must gate traffic — liveness must not depend on external DB (use separate health groups in Spring Boot 3).
• ConfigMap change = rolling restart — automate in GitOps pipeline; document which keys require restart.
• Kubernetes is not your CI — build image in CI, K8s pulls and runs; no kubectl apply from laptop to prod.

Interview questions

Hands-on exercise

Lab: Kubernetes YAML concepts:

• Run playground — review probe and memory checklist output.
• Sketch Deployment with replicas=3 and payment-service labels.
• Write readinessProbe path for Spring actuator.
• Map ConfigMap key to SPRING_KAFKA_BOOTSTRAP_SERVERS env.
• Bonus: explain rolling update with maxUnavailable=0.

Press Run to execute. Real javac + JVM via remote API. Falls back to simulator on network failure.

Architecture trade-offs

• Deployment vs StatefulSet: Deployment wins stateless Java APIs; StatefulSet wins embedded identity (Kafka broker).
• ClusterIP vs Ingress: ClusterIP wins internal; Ingress wins HTTP routing and TLS termination.
• ConfigMap mount vs env: Mount wins large yaml; env wins simple key-value.
• Eureka vs K8s DNS: K8s DNS wins on cluster; Eureka wins multi-cluster legacy.

Summary

Kubernetes runs Java microservices at fleet scale — Deployments for rollouts, Services for discovery, ConfigMaps and Secrets for externalized config. Wire Spring Boot actuator probes and graceful shutdown before claiming production-ready. Next: OpenTelemetry for observing requests across these pods.

Key takeaways

• Pod = run unit; Deployment = desired replicas + rolling updates.
• Service = stable DNS load balancer to ready pods.
• ConfigMap for config; Secret for credentials — never in image.
• Spring actuator liveness/readiness probes — gate traffic correctly.
• Memory limit + MaxRAMPercentage — coordinate JVM and K8s resources.