System Design for Java Engineers

Introduction

System design is how Java engineers turn business requirements into architectures that survive Black Friday, regional outages, and decade-long maintenance. When a staff engineer says "we need 99.99% availability with read-heavy traffic," they are translating product language into concrete decisions: horizontal scale, load balancers, read replicas, cache tiers, and explicit CAP trade-offs.

This lesson connects Java/Spring Boot services to the non-functional requirements every enterprise system must meet:

Scalability — handle 10× traffic without rewriting. Availability — survive node, zone, or dependency failure. Reliability — correct behavior under retries, partial failure, and chaos. CAP theorem — choose consistency vs availability during partitions. Load balancing — distribute traffic healthily across JVM instances.

You will learn to reason about these forces in architecture reviews and staff interviews — with real case studies from Netflix, Amazon, and Stripe.

Business problem

Teams that treat system design as "draw boxes in an interview" pay in production:

• Outages at peak: Single Spring Boot instance behind one ALB with no autoscaling — Prime Day traffic takes checkout offline for 47 minutes.
• Silent degradation: No health checks on JVM pods — load balancer keeps routing to OOM-killed instances; p99 latency climbs while CPU dashboards look fine.
• Wrong CAP choice: Strong consistency on a global product catalog during regional network partition — writes block globally; revenue stops in healthy regions.
• Reliability gaps: Retry storms from clients without idempotency keys — duplicate charges when payment service times out and callers retry.
• Cost explosions: Over-sharded microservices for 500 QPS — 40 JVM clusters where 3 autoscaling groups would suffice.

Why this topic exists

System design exists because software fails at scale unless forces are named and managed:

• Scale math: 10M DAU × 20 requests/day ≠ one Postgres instance — estimation drives architecture before code.
• Failure is normal: Disks, networks, and deployments fail — availability design assumes failure, not prevents it.
• Consistency is expensive: CAP forces explicit choices — financial ledgers vs social feed likes need different models.
• Java at scale: JVM heap, GC pauses, and thread pools become bottlenecks — load balancing and horizontal scale are first-line mitigations.
• Interview signal: Staff loops test whether you quantify trade-offs — not whether you memorized Netflix's diagram.

Core concepts

Core definitions — enterprise Java context:

• Scalability: System handles increased load by adding resources (horizontal: more pods; vertical: bigger machines). Measure: QPS, storage growth, bandwidth.
• Availability: Percentage of time system is usable — 99.9% = 8.76 hrs downtime/year; 99.99% = 52 min. Multi-AZ, health checks, graceful degradation.
• Reliability: System performs correctly over time — idempotent APIs, retries with backoff, circuit breakers, chaos testing. Availability without correctness is useless.
• CAP theorem: During network partition, choose Consistency (CP) or Availability (AP). CA only when partition impossible (single node). Most Java microservices are AP with eventual consistency on reads.
• Load balancing: Distribute requests across healthy backends — L4 (TCP), L7 (HTTP path routing), algorithms: round-robin, least connections, consistent hash for sticky sessions/cache affinity.

Internal architecture

Reference architecture — Spring Boot microservice at scale:

                    ┌─────────────── CDN (static assets) ───────────────┐
                    │                                                    │
  Clients ──HTTPS──▶│  L7 Load Balancer (ALB / NGINX / Envoy)           │
                    │    ├─ /api/orders  → order-service (3+ AZ)         │
                    │    ├─ /api/catalog → catalog-service (read-heavy)  │
                    │    └─ health: /actuator/health                     │
                    └────────────────────┬───────────────────────────────┘
                                         │
              ┌──────────────────────────┼──────────────────────────┐
              ▼                          ▼                          ▼
     order-service pods           catalog-service pods         payment-service
     (stateless JVM)              (cache-aside Redis)          (CP — strong consistency)
              │                          │                          │
              ▼                          ▼                          ▼
     PostgreSQL primary           Redis cluster +               PostgreSQL
     + read replicas             read replicas                 (single-writer)
              │
              ▼
     Kafka (async events — AP path for notifications, analytics)

CAP snapshot:
  • Catalog browse during partition → AP (stale cache OK)
  • Payment capture during partition → CP (fail closed, no double charge)

System design forces — five diagrams for architecture reviews:

Code walkthrough

Spring Boot production patterns — health checks, idempotency, and read scaling:

• Health probes: Liveness vs readiness — readiness removes pod from LB during startup or DB migration.
• Idempotency-Key: Standard pattern for reliable POST under client retries — Stripe, Adyen, internal APIs.
• Read replicas: @Transactional(readOnly=true) + routing datasource sends read traffic off primary.
• Graceful degradation: AP path when cache fails — slower response beats hard error on product browse.

// ═══════════════════════════════════════════════════════════════
// 1. Load balancer health — Spring Boot Actuator
// ═══════════════════════════════════════════════════════════════
// application.yml
// management.endpoints.web.exposure.include=health,info
// management.endpoint.health.probes.enabled=true
// → Kubernetes/ALB uses /actuator/health/liveness and /readiness

// ═══════════════════════════════════════════════════════════════
// 2. Idempotent payment — reliability under retries
// ═══════════════════════════════════════════════════════════════
@RestController
public class PaymentController {
    private final PaymentService paymentService;
    private final IdempotencyStore idempotencyStore;

    @PostMapping("/payments")
    public ResponseEntity<PaymentResult> pay(
            @RequestHeader("Idempotency-Key") String key,
            @RequestBody PaymentRequest req) {

        return idempotencyStore.find(key)
            .map(existing -> ResponseEntity.ok(existing))
            .orElseGet(() -> {
                PaymentResult result = paymentService.charge(req); // CP: transactional
                idempotencyStore.save(key, result);
                return ResponseEntity.ok(result);
            });
    }
}

// ═══════════════════════════════════════════════════════════════
// 3. Read scaling — CQRS-lite with @Transactional(readOnly=true)
// ═══════════════════════════════════════════════════════════════
@Service
public class CatalogQueryService {
    @Transactional(readOnly = true)
    @Cacheable(value = "products", key = "#sku")
    public ProductView getProduct(String sku) {
        return productReadRepo.findBySku(sku); // routed to read replica
    }
}

// ═══════════════════════════════════════════════════════════════
// 4. Availability — graceful degradation when cache down
// ═══════════════════════════════════════════════════════════════
@Service
public class ResilientCatalogService {
    public ProductView getProduct(String sku) {
        try {
            return cache.get(sku);
        } catch (CacheException e) {
            log.warn("Cache unavailable — falling back to DB", e);
            return db.get(sku); // slower but available (AP)
        }
    }
}

Production example

AWS production stack — Java order service at 5k QPS:

• Topology spread: Pods across AZ-a/b/c — survives single zone loss (availability).
• HPA: Autoscale 12→40 pods on CPU — horizontal scalability without manual ops.
• Least connections: Better than round-robin when checkout requests vary 200ms–3s.
• Drain delay: Reliability during rolling deploy — no dropped in-flight requests.

# Kubernetes Deployment — 3 AZ, HPA on CPU 70%
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
spec:
  replicas: 12
  template:
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
      containers:
        - name: order-service
          image: order-service:2.4.1
          resources:
            requests: { cpu: "500m", memory: "1Gi" }
            limits:   { cpu: "2", memory: "2Gi" }
          livenessProbe:
            httpGet: { path: /actuator/health/liveness, port: 8080 }
          readinessProbe:
            httpGet: { path: /actuator/health/readiness, port: 8080 }

---
# ALB target group — least connections for long checkout requests
# stickiness: disabled (stateless JWT sessions)
# deregistration delay: 30s (drain in-flight during deploy)

Enterprise case study

Netflix — availability and scalability at global scale: Netflix serves 200M+ subscribers from AWS with a "fail fast, recover gracefully" philosophy. During the 2012 Christmas Eve outage, a missing region failover path took streaming offline for hours — catalyzing their chaos engineering program (Chaos Monkey) and multi-region active-active architecture. Today, Netflix's Java microservices run stateless behind Eureka/Envoy load balancing with regional isolation: a partition in us-east-1 does not take down eu-west-1. Their CAP choice: video metadata and recommendations are AP (stale rows OK briefly); billing integrations are CP (fail closed).

• Before: Single-region dependency cascade — one Cassandra cluster issue → global outage.
• Decision: Multi-region with bulkheads; chaos testing in production; AP for browse, CP for billing.
• Result: 99.99% streaming availability; regional blast radius contained; Simian Army validates resilience weekly.
• Lesson for Java teams: Stateless Spring Boot + regional LB + explicit CAP per bounded context — not one consistency model for everything.

Performance considerations

Scalability performance — JVM-specific:

• Horizontal > vertical: 4× 2GB JVM pods outperform 1× 8GB pod — GC pause scales with heap; smaller heaps = shorter STW.
• Connection pooling: Each pod holds DB pool connections — 40 pods × 20 connections = 800 DB connections; size pools and use PgBouncer.
• Cache hit ratio: 95% cache hit on catalog reads → 20× effective read capacity without more DB replicas.
• LB overhead: L7 routing adds ~1–3ms — negligible vs 50ms DB query; use L7 for path-based routing to services.

Security considerations

System design security intersections:

• TLS termination: LB terminates TLS — pods receive plain HTTP in private VPC; mTLS between services for zero-trust.
• Rate limiting at edge: WAF + LB rate limits protect JVM from DDoS before threads exhaust.
• Health endpoint exposure: /actuator/health public; /actuator/env locked down — info disclosure via misconfigured Actuator is a common CVE pattern.
• Idempotency store: Idempotency keys must be authenticated per tenant — prevent cross-tenant replay attacks.

Scalability considerations

Scaling decision framework:

• Stateless first: Session in Redis/JWT — any pod handles any request; enables linear horizontal scale.
• Shard when single DB saturates: >10k write QPS or >1TB hot data — partition by tenant_id or order_id hash.
• Async for slow paths: Email, analytics, search indexing via Kafka — decouples write latency from side effects.
• CDN for static: Product images, JS bundles — 90% bandwidth off origin servers.
• Read/write split: 100:1 read:write ratio → 3 read replicas + Redis cache before sharding writes.

Production challenges

Common system design failures in Java enterprises:

• Sticky sessions on LB: Breaks autoscaling — new pods receive no traffic until session expires.
• No readiness probe: Deploy sends traffic to JVM still starting — 503 storm during every release.
• Retry without backoff: Client retries × 100 pods = retry storm — exponential backoff + jitter mandatory.
• Single CAP model: Strong consistency on social feed — p99 latency 2s; wrong tool for the domain.
• Monitoring availability only: 200 OK but wrong balance displayed — reliability requires business metric checks.

Common mistakes

• Designing for 1M QPS when product has 500 QPS — operational burden kills velocity.
• Ignoring GC impact when scaling vertically — 32GB heap with G1 still pauses under load.
• Using round-robin for WebSocket or long-polling — need least connections or dedicated connection LB.
• Assuming multi-AZ = multi-region — AZ failure handled; region failure needs explicit DR design.
• Treating CAP as "pick two forever" — consistency model is per operation, not per system.

Debugging guide

Diagnose scalability and availability incidents:

• LB uneven distribution: Check stickiness config, connection reuse, pod readiness — one pod at 100% CPU while others idle.
• Retry storm: Trace idempotency key usage; graph retry rate vs error rate — spike after timeout increase.
• Read replica lag: User sees stale data after write — measure replication lag; route read-your-writes to primary.
• Partition behavior: During network incident, which services reject vs degrade — CAP choice validation in postmortem.

# Check ALB target health
aws elbv2 describe-target-health --target-group-arn $TG_ARN

# Kubernetes pod distribution across AZ
kubectl get pods -o wide -l app=order-service

# JVM thread pool exhaustion (Micrometer)
curl localhost:8080/actuator/metrics/tomcat.threads.busy

Best practices

• Estimate QPS, storage, bandwidth before choosing components — show math in every design doc.
• Deploy stateless Spring Boot across 3+ AZ with readiness probes and pod topology spread.
• Choose CAP per bounded context — CP for money, AP for catalog and analytics.
• Implement Idempotency-Key on all mutating APIs called with retries.
• Use least connections or consistent hash LB algorithms matched to workload shape.
• Load test at 2× expected peak before launch — find bottleneck before customers do.
• Define SLOs (availability, latency) and error budgets — architecture serves SLOs, not the reverse.

Anti-patterns

• Single giant JVM: Vertical scale only — GC pauses and deploy blast radius grow unbounded.
• Database as session store via sticky LB: Couples scale to session affinity — use Redis or JWT.
• Global strong consistency: Spanner/Single-writer everywhere for a blog app — cost and latency unjustified.
• Health check = TCP only: Port open but app dead — use HTTP readiness with DB dependency check.
• Retry infinite loop: No max retries, no backoff — amplifies outages.

Staff engineer notes

• Staff engineers quantify: "99.9% availability requires multi-AZ; 99.99% requires multi-region or rapid failover — here's the cost delta."
• CAP is not academic — name your partition behavior in every ADR: "During split, order service rejects writes (CP)."
• Load balancing is the cheapest scalability win — most Java teams under-invest in health check quality and drain behavior.
• Reliability > availability: 100% uptime returning wrong account balance is worse than 99.95% with correct ledger.
• Interview tip: always close with "at 10× traffic, the first bottleneck is X; phase 2 is Y" — shows operational thinking.

Interview questions

Hands-on exercise

Lab: Design document for order service

• Estimate scale: 50k orders/day, peak 5×, avg payload 1KB — compute write QPS and storage/year.
• Draw architecture: clients → LB → Spring Boot → Postgres + Redis + Kafka.
• Label CAP choice per component — CP for order write, AP for order status read cache.
• Specify LB algorithm and health check paths.
• Add idempotency flow for POST /orders with retry scenario.
• Write "at 10× traffic" phase-2: what breaks first and mitigation.

Press Run to execute. Real javac + JVM via remote API. Falls back to simulator on network failure.

Architecture trade-offs

• Strong consistency vs latency: CP paths add coordination cost — use only where business requires.
• Multi-region vs cost: 99.99% availability expensive — match redundancy to SLO, not ambition.
• Cache vs freshness: High hit ratio vs stale reads — TTL + invalidation on write.
• L4 vs L7 LB: L4 faster; L7 enables path routing and WAF — most Java microservices need L7.

Summary

System design for Java engineers means quantifying scale, naming CAP trade-offs, and wiring reliability into every mutating API. You can now sketch a production Spring Boot architecture with LB, read replicas, cache, and idempotency — and defend it in a staff interview with Netflix-scale case study lessons.

Key takeaways

• Scalability — horizontal stateless JVM pods behind autoscaling and LB.
• Availability — multi-AZ, health probes, graceful degradation, chaos testing.
• Reliability — idempotency, retries with backoff, correct behavior under failure.
• CAP — choose CP or AP per bounded context, not globally.
• Load balancing — match algorithm to request shape; drain during deploy.