Garbage Collection

Introduction

A payment service that pauses 800ms every GC cycle misses authorization SLAs — customers see timeouts while the JVM stops-the-world to reclaim heap. Garbage Collection (GC) is automatic memory management: unreachable Payment DTOs, settled transaction buffers, and orphaned cache entries are reclaimed so your ledger service doesn't OOM. Choosing the wrong collector or ignoring memory leaks turns GC from invisible infrastructure into a P1 incident.

This lesson covers modern collectors — G1 (Java 9+ default), ZGC, and Shenandoah — memory leak patterns in banking apps, and GC tuning for latency-sensitive payment paths. You will read GC logs, correlate pause times with p99 latency, and know when to escalate from flags to architecture changes.

GC tuning is operational engineering — staff engineers own p99 during settlement windows, not just "add more heap."

Business problem

GC failures hit revenue and compliance during peak banking hours:

• Stop-the-world pauses: Full GC during Black Friday — 2-second freeze, payment gateway timeouts, $1.2M abandoned carts.
• Memory leak: ThreadLocal holding PaymentContext never cleared — heap grows 2GB/day until OOM after 72 hours uptime.
• Wrong collector: Parallel GC on 32GB heap — multi-second pauses unacceptable for real-time fraud scoring.
• Over-sized heap: 64GB heap "to avoid GC" — G1 pause times worse; recovery takes longer when leak finally triggers.

Why this topic exists

Manual memory management doesn't scale for enterprise Java — millions of short-lived objects per second in payment pipelines require automatic, tunable reclamation.

• Productivity: Engineers focus on ledger logic, not malloc/free per Payment object.
• Safety: GC eliminates most dangling pointer bugs — leaks are logical retention, not use-after-free.
• Throughput vs latency trade-off: Collectors optimize for batch throughput or sub-10ms pauses — pick for workload.
• Cloud economics: Right-sized heap + tuned GC = fewer pods — direct cost savings on K8s billing.

Core concepts

Five GC pillars for payment platform engineers:

• G1 GC: Default collector — divides heap into regions, targets pause-time goal (-XX:MaxGCPauseMillis), good general enterprise choice.
• ZGC: Sub-millisecond pauses on large heaps (Java 15+ production) — colored pointers, concurrent compaction; ideal for low-latency authorization.
• Shenandoah: Red Hat/Alpine concurrent collector — similar low-pause goals; alternative where ZGC not available.
• Memory leaks: Objects reachable but logically dead — static caches, ThreadLocal, unclosed listeners; heap fills, GC can't help.
• GC tuning: Heap sizing, pause goals, logging (-Xlog:gc*), JFR GC events — measure before changing flags.

Internal architecture

Heap regions and GC cycle in a payment JVM:

G1 Heap layout (simplified):
┌──────────────────────────────────────────────────┐
│ Eden regions  │ Survivor │ Old regions (tenured) │
│ (new Payment  │   S0/S1  │ (Account cache,       │
│  DTOs)        │          │  long-lived beans)    │
└──────────────────────────────────────────────────┘

Object lifecycle:
  new Payment() ──▶ Eden ──▶ survive GC ──▶ Survivor ──▶ Old gen
  unreachable ──▶ marked ──▶ reclaimed (stop-the-world or concurrent phase)

Collector selection (2026 enterprise):
  G1         — default, -XX:MaxGCPauseMillis=200, heaps 4-32GB
  ZGC        — -XX:+UseZGC, heaps large, pause < 10ms target
  Shenandoah — -XX:+UseShenandoahGC, similar niche

GC logging (Java 11+):
  -Xlog:gc*:file=gc.log:time,uptime,level,tags

Key metrics:
  Pause time p99     — correlate with payment API p99
  Allocation rate    — bytes/sec in Eden — drives GC frequency
  Promotion failure  — survivor can't move to old — full GC risk

Five concepts — each ties to a production GC incident pattern in financial services.

1. G1 GC

• Default since Java 9: Start with G1 for Spring Boot payment services — 4–32GB heap, -XX:MaxGCPauseMillis=200.
• Humongous objects: Arrays > half region size (often 512KB–1MB) — special path; large settlement batch arrays may trigger frequent humongous allocation.
• Mixed GC: Collects portions of Old gen concurrently with Young — avoids monolithic Full GC when tuned correctly.
• When G1 struggles: Heap > 32GB with strict <10ms pause — evaluate ZGC; chronic Full GC — suspect leak not collector.

2. ZGC

• Enable: -XX:+UseZGC (Java 21 LTS — production ready); generational ZGC in Java 21+ improves young gen efficiency.
• Use case: Real-time fraud scoring, low-latency card authorization — p99 pause < 10ms on 16–64GB heap.
• Trade-off: Slightly higher CPU for concurrent work — acceptable when latency SLA is revenue-critical.
• Monitoring: JFR ZGCPhasePause — validate pauses in staging before Black Friday cutover.

3. Shenandoah

• Enable: -XX:+UseShenandoahGC — bundled in OpenJDK; Red Hat builds common in enterprise Linux.
• vs ZGC: Both low-latency; choose based on JDK vendor support, benchmark on your payment workload, ops team familiarity.
• When to consider: G1 pause spikes on large Old gen; ZGC not approved by platform team — Shenandoah middle path.
• Container aware: Combine with -XX:MaxRAMPercentage=75.0 — don't over-allocate heap vs K8s limit.

4. Memory Leaks

• ThreadLocal leak: Pool thread retains PaymentContext with PAN after request — clear in finally; use try-with-resources wrapper.
• Unbounded cache: ConcurrentHashMap as dedup cache without TTL — every txn ID forever; use Caffeine with expireAfterWrite.
• Listener registration: Register for settlement events, never unregister — old beans retained across hot reload.
• Classloader leak: Metaspace + heap references to old ClassLoader — OOM after repeated deploys without restart.

5. GC Tuning

• Heap sizing: -Xms = -Xmx avoids resize pause; size for steady-state + 30% headroom — not maximum possible.
• Container limits: -XX:MaxRAMPercentage=75.0 — leave room for metaspace, thread stacks, native memory.
• GC logging: -Xlog:gc*,safepoint:file=gc.log:time,level,tags — parse with GCViewer or GCEasy.
• When NOT to tune: Rising old gen with flat traffic — fix leak first; tuning collector on leaking JVM delays OOM, doesn't fix root cause.

Code walkthrough

Memory leak demo — unbounded cache vs Caffeine TTL; GC logging observation:

• Unbounded cache: Classic banking leak — dedup map grows with every txn; add TTL and max size.
• ThreadLocal.remove(): Mandatory in finally/close on pooled threads — Tomcat worker retains PAN otherwise.
• System.gc(): Hint only — never rely in production; fix retention instead.
• Heap metrics: Runtime.totalMemory - freeMemory approximates used — Micrometer jvm.memory.used is production-grade.
• Caffeine: Replace hand-rolled eviction with expireAfterWrite(24, HOURS) for receipt cache.

import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicLong;

// BAD: unbounded static cache — memory leak in production
class BadPaymentCache {
    private static final ConcurrentHashMap<String, byte[]> CACHE = new ConcurrentHashMap<>();

    static void store(String txnId, byte[] receipt) {
        CACHE.put(txnId, receipt); // never evicted — heap grows forever
    }
    static int size() { return CACHE.size(); }
}

// GOOD: bounded cache with eviction (concept — use Caffeine in production)
class BoundedPaymentCache {
    private final ConcurrentHashMap<String, byte[]> cache = new ConcurrentHashMap<>();
    private final int maxSize = 10_000;

    void store(String txnId, byte[] receipt) {
        if (cache.size() >= maxSize) {
            cache.keySet().stream().findFirst().ifPresent(cache::remove);
        }
        cache.put(txnId, receipt);
    }
    int size() { return cache.size(); }
}

// ThreadLocal leak pattern — ALWAYS remove in finally
class PaymentContext implements AutoCloseable {
    private static final ThreadLocal<PaymentContext> CTX = new ThreadLocal<>();

    static PaymentContext open(String txnId) {
        PaymentContext ctx = new PaymentContext(txnId);
        CTX.set(ctx);
        return ctx;
    }
    static PaymentContext current() { return CTX.get(); }

    private final String txnId;
    PaymentContext(String txnId) { this.txnId = txnId; }

    @Override public void close() { CTX.remove(); } // prevent pool thread leak
}

class GcDemo {
    public static void main(String[] args) throws Exception {
        AtomicLong allocated = new AtomicLong(0);

        // Simulate leak — 50k receipts in unbounded cache
        for (int i = 0; i < 50_000; i++) {
            BadPaymentCache.store("TXN-" + i, new byte[1024]); // 1KB each
            allocated.addAndGet(1024);
        }
        System.out.println("Bad cache size: " + BadPaymentCache.size());
        System.out.println("Approx bytes retained: " + allocated.get());

        // ThreadLocal correct usage
        try (PaymentContext ctx = PaymentContext.open("PAY-001")) {
            System.out.println("Processing: " + PaymentContext.current().txnId);
        } // close() removes ThreadLocal

        // Suggest GC (not guaranteed) — for demo only
        System.gc();
        Runtime rt = Runtime.getRuntime();
        System.out.printf("Heap used/max MB: %d / %d%n",
            (rt.totalMemory() - rt.freeMemory()) / 1024 / 1024,
            rt.maxMemory() / 1024 / 1024);
    }
}

Production example

Production JVM flags — payment authorization service (Java 21):

• Xms = Xmx: Avoid heap resize STW during ramp-up — critical for payment pods under HPA.
• GC log rotation: filecount/filesize — disk full on long-running settlement batch nodes without rotation.
• HeapDumpOnOOM: Capture leak snapshot — analyze with Eclipse MAT; PCI: secure dump storage.
• Caffeine stats: recordStats + hit rate — detect cache misuse before heap impact.

# Kubernetes deployment — env JAVA_OPTS

# G1 default (general payment API)
JAVA_OPTS="
  -Xms4g -Xmx4g
  -XX:+UseG1GC
  -XX:MaxGCPauseMillis=200
  -XX:+ParallelRefProcEnabled
  -XX:MaxRAMPercentage=75.0
  -Xlog:gc*:file=/var/log/gc.log:time,uptime,level,tags:filecount=5,filesize=20m
  -XX:+HeapDumpOnOutOfMemoryError
  -XX:HeapDumpPath=/dumps
"

# ZGC variant (low-latency fraud scoring)
JAVA_OPTS_ZGC="
  -Xms8g -Xmx8g
  -XX:+UseZGC
  -XX:+ZGenerational
  -Xlog:gc*:file=/var/log/gc.log:time,level,tags
"

# Spring Boot Actuator + Micrometer alerts
management.metrics.enable.jvm=true
# Alert: jvm.gc.pause max > 500ms for 5min during business hours

# Caffeine cache — bounded payment status
@Bean
public Cache<String, PaymentStatus> statusCache() {
    return Caffeine.newBuilder()
        .maximumSize(100_000)
        .expireAfterWrite(Duration.ofHours(24))
        .recordStats()
        .build();
}

Enterprise case study

Incident 1 — National bank mobile payments Full GC cascade (2023): Authorization service ran Java 11, G1, 24GB heap with no -Xms matching -Xmx. After weekend deploy, traffic doubled Monday morning. Allocation rate spiked from settlement file replay. G1 couldn't meet 200ms pause goal — mixed GC fell behind, Old gen filled, consecutive Full GCs each pausing 4–8 seconds. Mobile app showed "Payment failed" for 23 minutes. APM correlated exactly with GC pause spikes. Fix: set -Xms24g -Xmx24g, reduced receipt cache from unbounded HashMap to Caffeine 100k entries (discovered 6GB leak of PDF receipt bytes), increased InitiatingHeapOccupancyPercent tuning after load test. Post-fix: max pause 180ms under peak.

Incident 2 — ACH processor OOM after 96 hours (2024): ThreadLocal PaymentContext stored on Tomcat virtual threads without remove() in filter finally block — each context ~2KB with full request payload. Pool of 500 threads × continuous load = slow heap climb. GC logs showed increasing GC time but no Full GC until hour 96 — then java.lang.OutOfMemoryError: Java heap space during Fed ACH window. Thread dump + heap dump revealed 2.1M retained PaymentContext instances referenced from ThreadLocalMap. Fix: try-with-resources filter clearing ThreadLocal; weekly rolling restart until patch deployed; added Micrometer gauge on ThreadLocal map size via diagnostic bean.

• Incident 1 symptom: p99 authorization 8s — GC logs show "Pause Full (G1 Evacuation Pause)" repeated.
• Incident 1 root cause: Unbounded receipt cache + heap resize + G1 mixed GC lag — not insufficient CPU.
• Incident 2 symptom: Linear heap growth over days — classic leak signature in GC log slope.
• Incident 2 root cause: ThreadLocal without remove on servlet filter — invisible in functional tests.

Performance considerations

GC performance for payment workloads:

• Allocation rate: Reduce short-lived object churn — reuse buffers, stream large settlement files instead of loading to List.
• Object pooling caution: Pool only expensive objects (cipher suites) — pooling DTOs often hurts GC (long-lived garbage in Old gen).
• Humongous allocations: Split large byte[] into chunks below region size — or use off-heap/direct memory with care.
• Collector CPU: ZGC/Shenandoah use concurrent threads — budget CPU limits in K8s (don't set limit = request exactly).
• Parallel GC legacy: Still on Java 8 batch jobs — acceptable for offline settlement with multi-second pause tolerance, not online auth.

Security considerations

GC and security in payment systems:

• Heap dump sensitivity: OOM dumps contain PAN/CVV in live objects — encrypt at rest, restrict access, redact before sharing.
• GC log paths: Logs may include class names revealing internal architecture — protect like application logs.
• Memory not zeroed immediately: Cleared object memory may linger before reuse — don't rely on GC for crypto key erasure; use explicit zeroing or off-heap HSM.
• DoS via allocation: Attacker triggers huge in-memory aggregation — GC thrashing; cap request payload and result set size.

Scalability considerations

GC at fleet scale:

• Per-pod heap: Smaller heaps (4GB) × more pods — shorter G1 pauses vs few 64GB monoliths.
• Horizontal scale: More pods increase total GC work cluster-wide — optimize allocation first.
• ZGC on large shared nodes: Fraud engine on 32GB single node — ZGC justifies large heap without pause penalty.
• Rolling restart policy: Even without leaks, weekly restart clears drift — belt-and-suspenders for ThreadLocal-heavy code.

Production challenges

Real GC production failures:

• Mistaking GC for slow SQL: Both raise p99 — GC logs distinguish STW pause from JDBC wait.
• Tuning without baseline: Random flag changes from blog posts — worse pauses; one change + load test.
• Ignoring native memory: Heap healthy but process OOM-killed — direct buffers, metaspace, thread stacks exceed container limit.
• Humongous object storm: Settlement batch loads 500MB JSON to byte[] — G1 humongous region churn; stream parse instead.
• GC log disk full: Silent stop logging — miss incident evidence; rotate and monitor disk.

Common mistakes

• Setting -Xmx to container memory limit — no room for native/off-heap; OOMKilled without Java heap OOM.
• Using System.gc() in application code — triggers STW; CMS/G1 may honor explicitly in some configs.
• Unbounded in-memory dedup/cache on payment singleton — leak masquerading as "need more heap."
• ThreadLocal on Tomcat/executor threads without remove() — slow leak over days.
• Switching to ZGC without load test — CPU throttling in K8s may hurt more than G1 pauses.

Debugging guide

Debug GC and memory issues:

• GC logs: Parse with GCEasy.io or GCViewer — pause distribution, allocation rate, Full GC frequency.
• jcmd GC.heap_info: Current heap usage and collector stats live.
• Heap dump on OOM: MAT Dominator Tree — find largest retained set (often static map or ThreadLocal).
• JFR: jdk.GCPhasePause, ObjectAllocationInNewTLAB — correlate pause with allocation hotspots.
• Native memory: jcmd VM.native_memory summary — when heap fine but RSS grows.

# Live GC info
jcmd $(pgrep -f payments.jar) GC.heap_info

# Heap dump (careful — contains PAN data)
jcmd <pid> GC.heap_dump /secure/dumps/heap.hprof

# GC log analysis one-liner — Full GC count
grep -c "Pause Full" /var/log/gc.log

# JFR GC recording
jcmd <pid> JFR.start settings=profile
jcmd <pid> JFR.dump filename=gc-profile.jfr

# Native memory tracking (start with flag)
-XX:NativeMemoryTracking=summary
jcmd <pid> VM.native_memory summary

Best practices

• Enable rotated GC logging in every production payment service — -Xlog:gc* with filecount.
• Set -Xms equal to -Xmx — eliminate heap resize pauses during traffic ramp.
• Use Caffeine or bounded caches with TTL — never unbounded ConcurrentHashMap for txn data.
• Always ThreadLocal.remove() in finally on pooled threads — servlet filters, executor callbacks.
• HeapDumpOnOutOfMemoryError to secure path — analyze leaks with MAT, not guess at flags.
• Load test GC at peak settlement volume before Black Friday — measure pause p99, not average.
• Fix memory leaks before tuning collector — rising Old gen occupancy is leak until proven otherwise.

Anti-patterns

• "Just add heap" without leak analysis — delays OOM, increases pause times, wastes K8s budget.
• Copy-paste GC flags from different JDK major version — ZGC flags changed Java 15→21.
• Caching entire Payment entity with lazy collections forever — Hibernate proxy + unbounded cache = leak.
• Ignoring MaxRAMPercentage in containers — heap equals limit, native OOMKill.
• Disabling GC logs in prod to save disk — fly blind during authorization SLA incident.

Staff engineer notes

• GC pause spike on APM graph lining up with STW in gc.log is the smoking gun — not "mystery latency."
• Linear old gen growth over 72 hours with flat traffic is always a leak — don't tune G1 yet.
• ThreadLocal leaks survive code review because tests use few requests — only shows under sustained load.
• ZGC is not magic — allocation rate and leaks still matter; sub-ms pause on empty heap demo means nothing.
• Heap dumps are PCI events — treat path, access, and deletion with same rigor as database backups.

Interview questions

Hands-on exercise

Lab: GC and memory retention

• Run playground — observe BadPaymentCache size and heap used after 50k entries.
• Replace BadPaymentCache with bounded version — compare heap after same loop.
• Run with -Xlog:gc*:stdout — note GC events after allocation loop.
• Demonstrate ThreadLocal leak: set without remove on loop threads — inspect with heap dump tool if available.
• Fix PaymentContext with try-with-resources — verify remove in finally.
• Bonus: run same app with -XX:+UseZGC (Java 21) vs G1 — compare pause times under load generator.

Press Run to execute. Real javac + JVM via remote API. Falls back to simulator on network failure.

Architecture trade-offs

• G1 vs ZGC: G1 simpler default; ZGC for strict pause SLA on large heap.
• Large heap vs more pods: Large heap = longer recovery from leak; small pods = shorter pauses, more GC aggregate work.
• In-memory cache vs Redis: Cache fast but leak risk per pod; Redis adds latency but bounded cluster memory.
• Throughput vs low-latency collector: Parallel GC for batch settlement; ZGC for online authorization.

Summary

Garbage collection keeps payment JVMs alive — but the wrong collector, unbounded caches, or ThreadLocal leaks turn GC into an SLA killer. Master G1 as default, evaluate ZGC/Shenandoah for low-latency authorization, read GC logs, and fix retention before tuning flags. Staff engineers treat Full GC during peak settlement as a paging event — with logs, heap dumps, and a leak-first mindset.

Key takeaways

• G1 is the default enterprise collector — tune MaxGCPauseMillis with GC logs and load tests.
• ZGC and Shenandoah target sub-10ms pauses — evaluate for latency-critical payment paths on Java 21.
• Memory leaks are reachable dead logic — unbounded caches and ThreadLocal without remove are top banking culprits.
• Always -Xms=-Xmx, MaxRAMPercentage, rotated GC logs, and HeapDumpOnOutOfMemoryError in production.
• Correlate GC pause timestamps with APM p99 — fix leaks before chasing collector flags.