Multithreading

Introduction

Every Spring Boot payment service, Kafka consumer, and batch settlement job runs on threads — lightweight execution paths scheduled by the JVM on OS threads. When two threads read and write the same account balance without coordination, you get race conditions. When two threads wait on each other forever, you get deadlocks and a frozen production service at 2 AM.

This lesson teaches enterprise concurrency from the ground up: the thread lifecycle (NEW → RUNNABLE → BLOCKED/WAITING → TERMINATED), synchronization with locks and synchronized, how races corrupt financial data, how deadlocks form, and the production issues staff engineers debug with thread dumps, JFR, and Micrometer metrics.

Multithreading is prerequisite knowledge for Java 21 virtual threads — you must understand platform thread behavior, pinning, and shared-state hazards before adopting Loom at scale.

Business problem

Concurrency bugs are expensive, intermittent, and reputation-damaging:

• Double-spend race: Two transfer threads read balance=100, both deduct 80 — final balance wrong; audit trail inconsistent.
• Deadlocked payment gateway: Thread A holds lock on Account, waits for Ledger; Thread B holds Ledger, waits for Account — all workers frozen, timeouts cascade.
• Thread pool exhaustion: 200 platform threads blocked on slow DB — new requests queue until SLA breach; mistaken for "need more servers."
• Non-deterministic test passes: Race only under production load — bug ships because CI runs single-threaded.

Why this topic exists

Modern servers are inherently concurrent — multiple requests, multiple cores, shared caches and connection pools. Java exposes threads since 1.0; the JVM maps them to OS threads with a defined lifecycle and scheduling model.

• Throughput: Process many payments in parallel on multi-core hardware.
• Responsiveness: Background threads handle audit logging while HTTP thread returns response.
• Correctness requirement: Shared mutable state must be coordinated — otherwise "works on my laptop" fails at scale.
• Operational reality: On-call engineers use jcmd Thread.print, JFR, and APM thread metrics — not just reading code.

Core concepts

Five pillars of enterprise multithreading:

• Thread lifecycle: NEW → RUNNABLE ↔ BLOCKED/WAITING/TIMED_WAITING → TERMINATED. Created via Thread, ExecutorService, or @Async.
• Synchronization: synchronized, ReentrantLock, volatile, atomic classes — mutual exclusion and visibility guarantees.
• Race conditions: Outcome depends on thread interleaving — check-then-act on shared balance without lock is classic bug.
• Deadlocks: Circular wait — each thread holds one lock and waits for another; JVM detects but recovery is painful.
• Production issues: Pool sizing, thread leaks, lock contention, false sharing, and confusing symptoms (CPU low but throughput zero = deadlock).

Internal architecture

Thread lifecycle and lock model:

Thread states (java.lang.Thread.State):
  NEW ──start()──▶ RUNNABLE ◀──▶ BLOCKED (waiting for monitor)
                      │              WAITING (wait/join)
                      │              TIMED_WAITING (sleep/timeout)
                      └──run() ends──▶ TERMINATED

Synchronization options:
  synchronized method/block     — intrinsic lock (monitor) on object
  ReentrantLock                 — explicit lock, tryLock, fairness
  volatile                      — visibility, not atomic compound ops
  AtomicInteger / LongAdder     — lock-free counters

Race condition pattern (BAD):
  if (balance >= amount) {   // Thread A and B both read 100
      balance -= amount;    // Both write — lost update
  }

Deadlock pattern (BAD):
  Thread-1: lock(A) → lock(B)
  Thread-2: lock(B) → lock(A)   // circular wait

Production stack:
  @Async / ExecutorService → bounded thread pool
  @Transactional           → DB connection per thread
  Micrometer               → thread pool active/queue metrics
  jcmd <pid> Thread.print  → thread dump for deadlock analysis

Five concepts — each diagram maps to a production failure mode you will debug on-call.

1. Thread Lifecycle

• NEW: Thread t = new Thread(task) — object exists, no OS thread yet until start() (never call run() directly).
• RUNNABLE: Eligible for CPU time — JVM scheduler picks among RUNNABLE threads on available cores.
• BLOCKED/WAITING: Blocked on synchronized lock; WAITING on wait(), join(); TIMED_WAITING on sleep() or lock timeout.
• TERMINATED: run() finished or uncaught exception — thread cannot restart; leak if pool never reclaims.

2. Synchronization

• synchronized: Intrinsic monitor lock — one thread enters block/method on same object at a time; others BLOCKED until release.
• ReentrantLock: Explicit API — tryLock(timeout) avoids indefinite deadlock; use finally { lock.unlock(); }.
• volatile: Guarantees visibility (read sees latest write) — does NOT make count++ atomic; use AtomicLong for counters.
• Prefer java.util.concurrent: ConcurrentHashMap, BlockingQueue, CountDownLatch — battle-tested over hand-rolled locks.

3. Race Conditions

• Definition: Correctness depends on scheduling order — bug may appear 1 in 10,000 runs under load.
• Check-then-act: if (balance >= amt) balance -= amt is NOT atomic — must synchronize entire operation or use atomic CAS.
• Compound actions: read-modify-write on shared fields needs lock, atomic variable, or immutable replacement.
• Fix pattern: synchronized void debit(amount) or single-thread executor for account mutations — serialize access per account ID.

4. Deadlocks

• Circular wait: Thread 1: lock accountA → lock accountB. Thread 2: lock accountB → lock accountA — neither proceeds.
• Prevention: Global lock ordering — always acquire locks in same ID order (lower account ID first).
• Detection: JVM thread dump shows Found one Java-level deadlock — jcmd <pid> Thread.print.
• tryLock timeout: Fail fast and retry/backoff instead of waiting forever — log alert for investigation.

5. Production Issues

• Thread pool exhaustion: All workers BLOCKED on DB — queue grows, p99 spikes; fix pool size, timeouts, or async I/O (virtual threads in Java 21).
• Thread leaks: new Thread() per task without pool — native memory and OS thread limit hit; use ExecutorService.
• Lock contention: Many threads on one synchronized method — CPU burn, low throughput; shrink critical section or partition locks by account ID.
• False sharing: Unrelated atomic counters on same cache line — rare but measurable; pad or use LongAdder.

Code walkthrough

Enterprise multithreading demo — lifecycle, sync, race fix, deadlock prevention:

• synchronized debit: Entire check-then-act atomic — prevents lost update race on balance.
• ExecutorService: Fixed pool reuses threads — avoids thread-per-request OS overhead and leaks.
• CountDownLatch: Coordinate simultaneous start — stress-test race conditions in tests.
• Lock ordering: TransferService acquires locks in consistent ID order — prevents AB-BA deadlock.
• Thread states: NEW before start(), TERMINATED after join() — never call run() for concurrency.

import java.util.concurrent.*;
import java.util.concurrent.locks.*;

// ── Shared account — synchronization required ──
class Account {
    private final String id;
    private long balanceCents;
    private final Object lock = new Object();

    Account(String id, long balanceCents) {
        this.id = id; this.balanceCents = balanceCents;
    }

    // Synchronized debit — prevents race on check-then-act
    synchronized boolean debit(long amountCents) {
        if (balanceCents >= amountCents) {
            balanceCents -= amountCents;
            return true;
        }
        return false;
    }

    synchronized long balance() { return balanceCents; }
    String id() { return id; }
}

// ── Deadlock-safe transfer — lock ordering by account ID ──
class TransferService {
    private final ReentrantLock lockA = new ReentrantLock();
    private final ReentrantLock lockB = new ReentrantLock();

    void transferOrdered(Account from, Account to, long amount) {
        Account first = from.id().compareTo(to.id()) < 0 ? from : to;
        Account second = first == from ? to : from;
        ReentrantLock l1 = first == from ? lockA : lockB;
        ReentrantLock l2 = second == to ? lockB : lockA;
        // Simplified demo — production: lock per account ID map
        l1.lock();
        try {
            l2.lock();
            try {
                if (from.debit(amount)) { /* credit 'to' */ }
            } finally { l2.unlock(); }
        } finally { l1.unlock(); }
    }
}

class MultithreadingDemo {
    public static void main(String[] args) throws Exception {
        Account account = new Account("ACC-1", 10_000L); // $100.00

        // ExecutorService — managed thread lifecycle
        try (ExecutorService pool = Executors.newFixedThreadPool(4)) {
            CountDownLatch start = new CountDownLatch(1);
            CountDownLatch done = new CountDownLatch(10);

            for (int i = 0; i < 10; i++) {
                pool.submit(() -> {
                    try {
                        start.await(); // all threads ready — maximize race if unsynchronized
                        account.debit(1_000L); // $10 each
                    } catch (InterruptedException e) {
                        Thread.currentThread().interrupt();
                    } finally {
                        done.countDown();
                    }
                });
            }
            start.countDown();
            done.await(5, TimeUnit.SECONDS);

            System.out.println("Final balance cents: " + account.balance());
            System.out.println("Expected: 0 (10 debits of 1000 from 10000)");
            System.out.println("Thread count active: " + Thread.activeCount());
        }

        // Thread lifecycle demo
        Thread t = new Thread(() -> {
            System.out.println("State in run: " + Thread.currentThread().getState());
        });
        System.out.println("Before start: " + t.getState()); // NEW
        t.start();
        t.join();
        System.out.println("After join: " + t.getState());   // TERMINATED
    }
}

Production example

Spring Boot — enterprise concurrency patterns:

• Bounded pools: core/max/queue prevent unbounded thread or queue growth under spike.
• @Transactional + threads: Do not pass Entity across threads — open new transaction in async worker.
• Lock striping: One lock per account ID — parallel transfers on different accounts don't block each other.
• Daemon threads: Audit workers don't prevent JVM shutdown — important for graceful K8s termination.

# application.yml — bounded async pool
spring:
  task:
    execution:
      pool:
        core-size: 8
        max-size: 32
        queue-capacity: 500
      thread-name-prefix: payment-async-

@Service
public class PaymentService {
    private final ExecutorService auditPool =
        Executors.newFixedThreadPool(4, r -> {
            Thread t = new Thread(r, "audit-" + System.nanoTime());
            t.setDaemon(true);
            return t;
        });

    @Transactional  // one DB connection bound to request thread
    public PaymentResult process(PaymentCommand cmd) {
        Payment payment = ledger.debit(cmd);
        auditPool.submit(() -> auditLog.record(payment)); // fire-and-forget
        return PaymentResult.ok(payment);
    }
}

// Per-account lock striping — reduce contention vs global lock
public class AccountLockRegistry {
    private final ConcurrentHashMap<String, Object> locks = new ConcurrentHashMap<>();
    Object lockFor(String accountId) {
        return locks.computeIfAbsent(accountId, k -> new Object());
    }
}

// Micrometer — monitor pool health
@Bean
public MeterBinder threadPoolMetrics(ExecutorService auditPool) {
    return registry -> registry.gauge("audit.pool.active", auditPool, ...);
}

Enterprise case study

E-commerce checkout double-charge incident: A retail platform processed payments with a if (cart.isPaid()) return; check followed by charge — without synchronization. Under Black Friday load, two HTTP threads both passed the check for the same cart ID within milliseconds. Both called the payment gateway — customer charged twice, 847 duplicate charges before circuit breaker triggered. Root cause: race condition on shared cart state in singleton service bean. Fix: synchronized block keyed by cartId via lock striping + idempotency key at gateway. Secondary finding: thread dump during incident showed 400/400 Tomcat threads BLOCKED on DB — pool sized for CPU not I/O latency.

• Symptom: Duplicate charges clustered at peak — not reproducible in single-thread QA.
• Thread dump: All workers BLOCKED on same JDBC connection pool — compounded latency.
• Fix 1: Per-cart lock + idempotent payment API with dedup key.
• Fix 2: Increased connection pool, added HikariCP leak detection, async audit off request thread.

Performance considerations

Concurrency performance trade-offs:

• Lock granularity: Coarse lock (one global) = simple but serializes everything; fine-grained (per account) = faster but deadlock risk if ordering wrong.
• synchronized vs ReentrantLock: Similar performance on HotSpot; ReentrantLock wins with tryLock, fairness, and Condition queues.
• LongAdder vs AtomicLong: High-contention counters — LongAdder stripes across cells, reduces cache line bouncing.
• Thread pool sizing: CPU-bound ≈ core count; I/O-bound historically higher — Java 21 virtual threads change this calculus.
• Context switching: Too many runnable threads on few cores — overhead dominates; right-size pools and measure.

Security considerations

Concurrency and security:

• TOCTOU races: Time-of-check-time-of-use — authorization check then action without lock allows privilege bypass in multi-threaded handlers.
• Session ThreadLocal: Request context in ThreadLocal — async thread doesn't inherit; can leak credentials if pool reuses threads without clear.
• Idempotency: Race on "create payment" must use idempotency keys — concurrency bug becomes financial fraud vector.
• DoS via thread exhaustion: Unbounded newCachedThreadPool() — attacker triggers unlimited thread creation.

Scalability considerations

Scaling concurrent services:

• Horizontal scale: Stateless services scale pods — but shared DB row still serializes; design for partitionable locks.
• Queue-based decoupling: Kafka/SQS between stages — each consumer pool sized independently.
• Read-write locks: ReentrantReadWriteLock when reads dominate (balance queries) and writes rare (debit).
• Virtual threads (Java 21): Platform multithreading fundamentals still apply to shared state — virtual threads don't remove need for locks.

Production challenges

Real production concurrency failures:

• Intermittent ClassCastException: Non-thread-safe HashMap in cache — use ConcurrentHashMap.
• Silent data corruption: Race without exception — only detected in reconciliation batch job days later.
• Deadlock under load only: Low traffic never hits lock order conflict — stress test with concurrent transfers.
• ThreadLocal memory leak: Pool + ThreadLocal + large objects — OOM after hours; clear in finally block.
• @Async on same bean: Self-invocation bypasses proxy — async never runs; extract to separate bean.

Common mistakes

• Calling thread.run() instead of start() — runs on caller thread, no concurrency.
• Synchronizing on wrong object — two instances still race if lock is not shared.
• Using volatile for count++ — visibility yes, atomicity no — use AtomicInteger.
• Unbounded CachedThreadPool in production — unbounded thread growth under spike.
• Holding lock during external HTTP call — blocks all other threads needing same lock.

Debugging guide

Debug concurrency in production:

• Thread dump: jcmd <pid> Thread.print or kill -3 <pid> — find BLOCKED threads and lock owners.
• Deadlock section: JVM prints "Found one Java-level deadlock" with stack traces — identify lock order.
• JFR: Enable lock profiling and thread park events — see contention hotspots.
• Reproduce races: CountDownLatch + many threads + ThreadLocalRandom seed — run in loop in test.
• AsyncMDC: Log thread name + trace ID — correlate which thread handled payment.

# Thread dump
jcmd $(pgrep -f payments.jar) Thread.print > threads.txt

# Look for BLOCKED and deadlock:
grep -A5 "deadlock\|BLOCKED" threads.txt

# JFR lock profiling (Java 11+)
jcmd <pid> JFR.start settings=profile
# ... reproduce ...
jcmd <pid> JFR.dump filename=locks.jfr

# Stress test pattern in JUnit
ExecutorService pool = Executors.newFixedThreadPool(32);
CountDownLatch latch = new CountDownLatch(1);
for (int i = 0; i < 1000; i++) pool.submit(() -> { latch.await(); service.transfer(); });
latch.countDown();

Best practices

• Prefer java.util.concurrent over raw Thread — ExecutorService, concurrent collections.
• Always use bounded thread pools in production — name threads for debugging (payment-worker-1).
• Synchronize or atomize entire check-then-act — never split read and write without coordination.
• Acquire locks in global order when multiple locks needed — prevents circular wait.
• Keep critical sections small — no I/O or external calls while holding lock.
• Use idempotency keys for payment operations — defense in depth against races.
• Thread dump first when service "hangs" with low CPU — likely deadlock or pool exhaustion.

Anti-patterns

• double-checked locking without volatile — broken before Java 5 memory model; use holder idiom or enum singleton.
• Synchronizing on String literals or boxed Integers — JVM may intern, unrelated code shares lock.
• Global synchronized on all API methods — throughput ceiling of one request at a time.
• Ignoring InterruptedException — break cooperative cancellation; restore interrupt flag.
• Sharing mutable static state across requests in Spring singleton — use request-scoped or immutable state.

Staff engineer notes

• Races don't throw exceptions — they corrupt data quietly; reconciliation jobs are often where concurrency bugs surface.
• Deadlocks are a design bug, not bad luck — fix lock ordering in code review when you see nested locks.
• Thread dumps are free and instant — learn to read them before reaching for heap dumps.
• Java 21 virtual threads simplify I/O concurrency but shared mutable state still needs locks — multithreading fundamentals unchanged.
• In fintech code review: any mutable field on a singleton @Service without synchronization or confinement is a red flag.

Interview questions

Hands-on exercise

Lab: Multithreading and correctness

• Run playground — observe synchronized debit leaving balance 0 after 10 concurrent debits.
• Remove synchronized from debit — rerun; note balance may be wrong (race).
• Print thread state before/after start and join in a small Thread demo.
• Add two Account locks and simulate deadlock (comment out lock ordering) — capture thread dump conceptually.
• Rewrite debit using ReentrantLock with tryLock(1, SECONDS) — log timeout instead of blocking forever.
• Bonus: configure a fixed thread pool of 2 and submit 5 tasks — observe queue behavior.

Press Run to execute. Real javac + JVM via remote API. Falls back to simulator on network failure.

Architecture trade-offs

• synchronized vs explicit locks: synchronized simpler; ReentrantLock for tryLock and fairness.
• Coarse vs fine locks: Coarse = safe/simple; fine = faster but deadlock risk.
• Thread pool vs per-task Thread: Pool wins production; virtual threads change I/O-bound calculus.
• Optimistic vs pessimistic locking: Optimistic (version column) wins low contention; pessimistic wins high conflict transfers.

Summary

Enterprise multithreading is about correct shared-state access under load — not just creating threads. Master the lifecycle, synchronize critical sections, understand races and deadlocks, and know how to triage production freezes with thread dumps and pool metrics. These fundamentals apply whether you use platform threads or Java 21 virtual threads. Next: Java 21 LTS for virtual thread concurrency at scale.

Key takeaways

• Thread lifecycle: NEW → RUNNABLE ↔ BLOCKED/WAITING → TERMINATED — use ExecutorService, not raw Thread.
• Synchronize entire check-then-act on shared state — volatile alone does not prevent races.
• Race conditions corrupt data without exceptions — test with CountDownLatch under load.
• Prevent deadlock with consistent lock ordering and tryLock timeouts.
• Production: thread dumps for hangs, bounded pools, lock striping, idempotency keys for payments.