Caching Strategies and Cache Invalidation

Advanced System Design & Software Architecture · Lesson 27 · High-Level Design and Distributed Systems

Lesson Overview

TechLearningPro's launch morning, the one lesson 24 sized and lesson 25 diagnosed, was survivable only if course outlines did not all hit the primary. The cache that made the miss rate small also served an outline from before a price change, and a second cache — added because the first one "worked" — served a remaining-seat count that was already wrong. Two learners bought the last seat. Finance saw one extra order. The instructor saw a cohort over the cap. The dashboard showed a hit ratio worth celebrating.

A cache is a copy with a permission slip. The slip says which fact may be copied, how old the copy may be, who is allowed to read it, and what happens when the source changes. Cache-aside, read-through, write-through, and write-behind are different slips. A TTL is a promise that staleness ends. An explicit delete is a promise that someone runs it after the commit, not before. A stampede is what happens when a popular key expires and every API node, the fleet lesson 26 just made stateless, refills it together. A hot key is the course everyone in the cohort opens, which one shard of a shared cache cannot serve no matter how good the hit ratio looks in aggregate.

/learn/high-level-design/cache-strategies and /learn/system-design/caching-layers name the patterns. /learn/high-level-design/redis introduces a cache server. This lesson does not invent commands for that server. It uses Spring's cache annotations, Caffeine's loading cache, and Spring Data Redis ValueOperations.setIfAbsent where a lock with a timeout is required. The decision is ADR-106: outlines may be stale for a bounded window; seat counts may not be cached at all. Lesson 2 already forbade a price cache from authorizing a charge. This lesson is where that rule becomes a key design.

Learning Objectives

By the end of this lesson you will be able to:

1. Match cache-aside, read-through, write-through, and write-behind to a freshness and durability requirement, and say what each one does when the cache process dies.
2. Set a TTL as a bounded staleness promise, and pair it with an invalidation that runs only after the source commit succeeds.
3. Explain a stampede and contain it inside one JVM with a loading cache, and across JVMs with a short-lived lock that cannot outlive a crashed holder.
4. Isolate a hot key with a small local cache whose age is inside the same staleness window.
5. Keep tenant and caller identity in the cache key, and keep private responses out of a shared cache.
6. Refuse to cache a fact whose scenario allows no staleness, such as remaining seats, and record that refusal.
7. Use HTTP cache directives for immutable media without letting a CDN store an authenticated outline.

Prerequisites

• Lesson 24's launch envelope: thousands of catalog reads per second at the burst, and a miss rate the primary can survive. This lesson is how that miss rate is achieved and how it lies.
• Lesson 25's serial outline hop. The cache exists to keep that hop off the critical path of most requests.
• Lesson 26's stateless API nodes. Nothing in this lesson may require a HashMap on one JVM as the only copy.
• Lesson 2's rule that a quote cache does not capture a payment, and lesson 10's ADR shape.
• Spring Framework cache annotations (@Cacheable, @CacheEvict, @CachePut) and, for the local tier, Caffeine. For a shared tier, Spring Data Redis ValueOperations. You do not need a custom Redis module, and this lesson will not speak in invented commands.

Detailed Core Concepts

A copy is a fact with an age

The source of an outline is the catalog primary. A cache entry is a projection of that outline at some time. Readers of the entry have agreed, whether they know it or not, to accept an outline that might be as old as the TTL or as old as the gap since the last successful delete. If product has not agreed, the cache is an incident in waiting. Write the agreement in the scenario before you pick a library: "after publish, 99 percent of outline reads observe the new price within 60 seconds." That sentence is implementable. "Always consistent" is not, once a copy exists.

The agreement has a security line too. The projection must not contain another tenant's rows, and it must not contain a caller's private notes. A shared cache key of courseId alone is wrong on a multi-tenant Academies cluster. The key is tenantId and courseId, and the value is the outline a member of that tenant may see. Personal progress is not in that value. Personal progress stays Cache-Control: private, no-store if it is ever an HTTP response.

Cache-aside

The application talks to the cache and to the source. On a read it looks up the key. On a miss it reads the source, then puts the value with a TTL. On a write it updates the source and then deletes the key. The cache is not on the write path as a system of record. If the cache is empty, the source still has the data. If the cache is wrong, the TTL or the delete bounds how long.

This is the pattern for outlines. Spring's @Cacheable is cache-aside implemented by a proxy: the method body is the source load and runs on a miss. @CacheEvict is the delete. The proxy will not save you from evicting before the transaction commits. That race is below, and it is the one that makes "we used the annotation" insufficient.

Cache-aside can stampede, because a miss is visible to every caller and every one of them may run the load. It can also cache a failure if you put a value on an error path you did not mean to. It does not, by itself, keep the source and the cache in one atomic action. The window between commit and delete is a window of staleness. Make it short and make it one-sided: readers may see the old outline, and they must not see a mix of old price and new title that your code assembled from two keys updated independently. One key per outline document avoids that mix.

Read-through

Read-through moves the load into the cache provider. The application asks the cache, and the cache, on a miss, calls a loader you configured. Caffeine's LoadingCache is this shape inside one process: get either returns the value or runs the loader. The application code does not contain the miss branch. The failure mode moves with it. If the loader is slow or throws, the cache is where you observe it. If you have many application processes, each has its own loading cache unless you picked a shared provider that supports a loader. Redis, used as a plain key-value store through Spring Data Redis, does not call your database for you. Treating it as read-through without a loader you actually wrote is how the miss path silently becomes cache-aside again, with none of the care.

Read-through is attractive when one process wants coalesced loads. It is not a reason to hide the database from the design. The database is still the source. The loader must use the same authorization as a direct read, or the cache becomes a way around tenant checks.

Write-through

Write-through updates the cache and the source before the caller is told the write succeeded. The usual order that keeps the source authoritative: write the database, then put the new value in the cache, then return. If the cache put fails, the write has still committed, and the next read might miss and reload — which is safe — or might see a stale entry you failed to replace. So a failed put must delete the key. Leaving the old entry in place after a successful commit is the bug write-through is supposed to prevent.

Write-through costs latency on the writer: the publisher waits for the cache. It helps when readers must observe the new value immediately and you control every writer. It does not help if another writer updates the database and skips the cache. There is one writer for a course outline: the catalog publish path. That is what makes write-through thinkable. It is still more machinery than cache-aside plus a short TTL, and ADR-106 chooses the latter because a 60-second window is acceptable and a synchronous cache dependency on publish is not.

Write-behind

Write-behind acknowledges the write when the cache has it, and copies to the source later. The cache is temporarily the only copy. A crash of the cache process loses the write. A reader who hits another region, or the database directly, does not see it. This pattern fits high-rate telemetry you can lose or rebuild, not a price and not a seat count. TechLearningPro's heartbeat stream, if it is buffered, belongs in a durable log with lesson 33's semantics, not in a cache that pretends to be a queue. This lesson names write-behind so you can refuse it for catalog facts. It does not implement it.

The refusal is the point. Write-behind looks like throughput in a diagram and like missing orders in an incident. If you cannot name the durability story — a replicated log, a replay, a bound on loss — you do not have write-behind. You have a bug.

TTL, explicit invalidation, and the race around commit

A TTL is the maximum age you will serve if every explicit delete fails. It is not a backup plan you hope to avoid. Design the product window to the TTL, then also delete on publish so the common case is faster than the worst case. A TTL of 60 seconds with a successful delete means most readers see the new outline in a moment. A failed delete means they see it within a minute. The scenario's measure is the minute, because that is the promise you can keep when the cache is sick.

Deleting before the commit is the classic race. A request misses, reads the old row, and the transaction then commits the new row. The cache now holds the old row with a fresh TTL. You have lengthened staleness to a full TTL at the exact moment of publish. Deleting before the write has the same shape: the miss reload observes uncommitted state, which under typical isolation is the old row, and stores it.

The delete belongs after the commit. Spring's @CacheEvict runs around the method, not around the transaction, unless you arrange the order. The reliable hook is TransactionSynchronization.afterCommit, or a @TransactionalEventListener in AFTER_COMMIT that performs the eviction. If there is no transaction, evict after the successful write returns. If the write rolls back, do not evict: the old cache entry is still correct.

Two keys for one document — outline price in one entry, outline title in another — double the race. Publish should refresh or delete one entry that holds the document the reader renders. Partial documents in separate keys are how a page shows the new title and the old price, which support cannot explain because no single row looks wrong.

Stampedes

A stampede, or dogpile, happens when a hot key misses and many requests load the source together. Causes: a TTL that expires on a cohort's schedule, a deploy that starts with an empty local cache, a delete that worked. The primary sees a burst equal to the concurrency of that key, which can be the whole launch even though the steady miss rate looked like 5 percent.

Inside one JVM, Caffeine's LoadingCache coalesces concurrent loads of the same key: one loader runs, waiters share the result. That is a documented property of its loading cache, and it stops a single process from multiplying the query. It does nothing about the other nineteen processes. Across processes, a short-lived lock is enough if it cannot stick forever. ValueOperations.setIfAbsent(key, value, timeout, unit) stores a marker only when absent and gives it a TTL, so a holder that crashes still releases the lock when the timeout passes. The winner loads the source and fills the cache. The others wait briefly and read the cache, or load themselves if the winner fails. The lock TTL is a few seconds, not the outline TTL. A long lock is an outage of that key.

Jitter the outline TTL by a small random amount so a fleet that filled together does not expire together. Jitter spreads a stampede. It does not replace the lock. refreshAfterWrite on Caffeine refreshes a key before it expires, which keeps a local hot key warm; the refresh still needs a single loader, which the loading cache provides inside that process.

@Cacheable(sync = true) asks the Spring cache provider to collapse concurrent loads. Support depends on the provider. It is a useful hint on a local Caffeine cache. It is not a cross-node lock on a shared Redis cache unless you have verified that provider's behavior for your version. This lesson uses the explicit setIfAbsent lock so the cross-node story is visible in our code, not implied by an attribute.

Hot keys

Aggregate hit ratio can be 98 percent while one course — the compliance course every Academies tenant assigned at 09:00 — accounts for most of the misses and most of the latency. That key lives on one slot of a shared cache. Adding cache nodes does not move it if the key still hashes to one of them. The practical containment is a small local cache in front, with a maximum size and a TTL inside the staleness window, so each API node absorbs the hot key in memory. Invalidation of the local tier is the short TTL, not a broadcast. A broadcast invalidation is a second system. Take it when the window cannot shrink further and you have operators for it. Until then, the local TTL is the bound, and it must be less than or equal to the promise.

A local cache that is large and long-lived reintroduces the sticky-heap bug from lesson 26 without the cookie: each node is warm for whoever it saw, and a drain throws that warmth away. Keep it small. The shared tier is the copy that survives a deploy.

What not to cache

Remaining seats, payment totals, authorization decisions for a specific learner, and one-time tokens do not go in the outline cache. Seats change when any enrollment commits. A cached count will oversell under the launch, which is the only time the count matters. Read the count from the primary inside the transaction that decrements it. The outline can say the course exists and what it costs, within the window. It cannot say "3 seats left" unless that string is decorative and the write path ignores it. Decorative counts will be screenshotted by learners and then treated as a promise. Omit them.

Negative caching — storing a "missing" marker so a bad id does not hammer the database — is legitimate for enumeration attacks on permalinks and dangerous for a course that was just created. A negative entry needs a short TTL, shorter than the positive one, and publish of a new course must delete the negative key. Spring's disableCachingNullValues refuses to store null. That avoids poisoning the cache with a null from a bug, and it also means a missing key is asked of the database every time. Choose per cache. Outlines disable null caching. A dedicated negative map, if you need one, is a different cache name with a TTL of a few seconds.

HTTP caches are a tier too

RFC 9110 and RFC 9111 define Cache-Control, and RFC 8246 defines immutable. Immutable media URLs from lesson 26 can be public, max-age=31536000, immutable because the URL changes when the bytes change. An outline response that varies by tenant and by authentication is private, no-store unless you have built a shared-cache key that includes the tenant and you have a surrogate policy you actually operate. A CDN in front of the API with a default TTL is a shared cache you do not control entry by entry. Prefer not to put authenticated outlines there. The application cache is enough for the launch QPS. The edge cache is for the video bytes.

ETag validators are appropriate when a client can tolerate a round trip that returns 304. They do not remove the need for the server-side decision. A 304 of a representation the user should no longer see is still a bug. Validators are a bandwidth tool. They are not an authorization tool.

Architecture and Design Diagrams

The outline read is allowed to miss the primary. The seat write is not allowed to miss it, and it never consults the outline cache.

mermaidCopy
flowchart LR
    client[Learner]
    api[API node]
    local[Local Caffeine cache]
    shared[Shared cache]
    db[(Catalog primary)]

    client --> api
    api --> local
    local -->|miss| shared
    shared -->|miss| db
    api -->|enroll, seats| db

Publish commits, then deletes. A reader during the gap may still hold the old document until the delete or the TTL.

mermaidCopy
sequenceDiagram
    participant P as Publish
    participant D as Primary
    participant C as Shared cache
    participant R as Reader

    P->>D: commit new outline
    D-->>P: committed
    P->>C: delete outline key
    R->>C: get
    alt deleted
        R->>D: load
        R->>C: put with TTL and jitter
    else gap before delete
        C-->>R: previous outline
    end

The stampede lock is per key, short, and separate from the outline entry.

mermaidCopy
flowchart TD
    miss[Shared miss on hot key] --> lock{setIfAbsent lock with TTL}
    lock -->|acquired| load[Load primary once]
    load --> fill[Put outline]
    fill --> unlock[Delete lock]
    lock -->|not acquired| wait[Short wait, then read cache]
    wait --> serve[Serve or load if still empty]

Technical Explanation

The outline document

One value contains the fields the lesson page can show without a second source: title, price, module list, and the published version. It does not contain remaining seats, the caller's progress, or another tenant's price list. The cache name is lesson-outline. The key is tenantId + ":" + courseId. A publish increments the version in the database and then deletes that one key.

The shared entry's TTL is 60 seconds plus up to 10 seconds of jitter, which matches a scenario measure of "new price visible within 70 seconds in the worst case, usually after the delete." The local Caffeine tier uses expireAfterWrite of 15 seconds and maximumSize of 10,000. Fifteen seconds is inside the 70-second promise. A node that never hears about the delete still sheds the old outline quickly. A node that hears about it, because we also evict locally in the same after-commit hook, sheds it immediately.

Why the eviction is in afterCommit

@CacheEvict on a @Transactional method can run while the transaction is still open, depending on advisor order. Another node can miss, read the old committed row, and refill. Doing the eviction inside afterCommit closes that particular race. It does not close the gap between commit and the delete call itself. Readers in that gap see the old value. The TTL bounds every gap, including a delete that throws.

If afterCommit runs and the cache client is down, the publish still succeeds. The outline is stale until the TTL. That is the availability choice: publish does not depend on the cache. The opposite choice — fail the publish if the cache cannot be cleared — couples a cache incident to content operations. ADR-106 accepts staleness over a blocked publish, because the window is short and the price is also re-read from the primary at capture time, as lesson 2 required. The outline cache is not an input to charging.

Seats

The enrollment transaction reads the seat count with a row lock, or updates a remaining column with a conditional update that succeeds only when the count is positive. It does not get the outline cache. A UI that wants to display a count reads it from the primary and marks the response no-store. At launch QPS that display can be dropped entirely. Omitting a live count is a product decision recorded next to the cache decision, so someone does not "just add it" to the outline document the night before a cohort start.

Penetration and poison

A bug that caches an empty outline for a real course, with a 60-second TTL, hides the course for a minute. Disable caching of null in the shared configuration. Do not catch a database exception and cache a fallback document; that stores the outage. On a loader failure, throw, and let the request fail. The local loading cache should not retain the exception as a long-lived value. Caffeine can be told not to store a failed load; the default loader behavior is to propagate, which is what we want.

Practical Implementation or Design Walkthrough

Problem statement

Serve catalog outlines at the launch burst with a bounded stale window. Never serve remaining seats from that cache. Keep a hot course from stampeding the primary across the API fleet.

Local tier

Caffeine's loading cache coalesces misses inside the node and refreshes a hot key before expiry. The loader calls the shared tier, not the database directly, so the database sees at most one shared fill per key.

javaCopy
package com.techlearningpro.catalog.cache;

import com.github.benmanes.caffeine.cache.Caffeine;
import com.github.benmanes.caffeine.cache.LoadingCache;
import java.time.Duration;
import java.util.Optional;

public final class LocalOutlineCache {

    private final LoadingCache<String, Optional<LessonOutline>> cache;

    public LocalOutlineCache(SharedOutlineCache shared) {
        this.cache = Caffeine.newBuilder()
                .maximumSize(10_000)
                .expireAfterWrite(Duration.ofSeconds(15))
                .refreshAfterWrite(Duration.ofSeconds(5))
                .build(shared::get);
    }

    public Optional<LessonOutline> get(String tenantId, String courseId) {
        return cache.get(cacheKey(tenantId, courseId));
    }

    public void evict(String tenantId, String courseId) {
        cache.invalidate(cacheKey(tenantId, courseId));
    }

    static String cacheKey(String tenantId, String courseId) {
        if (tenantId == null || tenantId.isBlank() || courseId == null || courseId.isBlank()) {
            throw new IllegalArgumentException("tenant and course are required");
        }
        return tenantId + ":" + courseId;
    }
}

Optional is the value so a missing course is not stored as a successful null that Spring might drop, and so a present outline is distinct. The shared tier still disables null caching for its own map. refreshAfterWrite keeps the hot key from expiring in a lump on a node that is actively reading it. A node that is quiet still uses expireAfterWrite.

Shared tier and the lock

The shared cache is Redis via Spring Data Redis. The outline entry uses ValueOperations.set with a TTL. The stampede lock uses setIfAbsent with a shorter TTL. Both methods are on ValueOperations. No command string is assembled in application code.

javaCopy
package com.techlearningpro.catalog.cache;

import java.time.Duration;
import java.util.Optional;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.data.redis.core.ValueOperations;

public class SharedOutlineCache {

    private static final Duration BASE_TTL = Duration.ofSeconds(60);
    private static final Duration LOCK_TTL = Duration.ofSeconds(5);

    private final RedisTemplate<String, LessonOutline> outlines;
    private final RedisTemplate<String, String> lockKeys;
    private final ValueOperations<String, String> locks;
    private final OutlineRepository repository;

    public SharedOutlineCache(
            RedisTemplate<String, LessonOutline> outlines,
            RedisTemplate<String, String> lockTemplate,
            OutlineRepository repository) {
        this.outlines = outlines;
        this.lockKeys = lockTemplate;
        this.locks = lockTemplate.opsForValue();
        this.repository = repository;
    }

    public Optional<LessonOutline> get(String key) {
        LessonOutline hit = outlines.opsForValue().get(key);
        if (hit != null) {
            return Optional.of(hit);
        }
        Boolean acquired = locks.setIfAbsent("fill:" + key, "1", LOCK_TTL.toSeconds(), TimeUnit.SECONDS);
        if (!Boolean.TRUE.equals(acquired)) {
            return Optional.ofNullable(outlines.opsForValue().get(key));
        }
        try {
            Optional<LessonOutline> loaded = repository.find(key);
            loaded.ifPresent(value -> outlines.opsForValue().set(
                    key,
                    value,
                    BASE_TTL.plusSeconds(ThreadLocalRandom.current().nextInt(0, 11))));
            return loaded;
        } finally {
            lockKeys.delete("fill:" + key);
        }
    }
}

If the lock is not acquired, the caller returns whatever is now in the cache, including empty. The API may retry once locally or accept a miss through to a single-flight on Caffeine's loader, which will call get again. What it must not do is ignore the lock and always hit the database. Boolean.TRUE.equals treats a null return, which a broken connection can produce, as "did not acquire," so a Redis failure does not look like a win. On that failure the method returns empty and the request fails rather than stampeding. A stricter policy can fall through to the database when Redis is down and accept the stampede as the price of a cache outage. ADR-106 chooses fail-the-read over a stampede: the primary stays up, and outlines return when the cache returns. That is a deliberate availability trade. If you would rather serve from the database when Redis is down, write that in the record and size the primary for a 100 percent miss at launch, which lesson 25 already showed does not fit.

The finally block deletes the lock with RedisTemplate.delete. If the process dies first, LOCK_TTL releases it. Do not use a lock without a TTL.

Evict after commit

javaCopy
package com.techlearningpro.catalog.cache;

import org.springframework.cache.Cache;
import org.springframework.cache.CacheManager;
import org.springframework.stereotype.Service;
import org.springframework.transaction.support.TransactionSynchronization;
import org.springframework.transaction.support.TransactionSynchronizationManager;

@Service
public class OutlinePublisher {

    private final OutlineRepository repository;
    private final CacheManager cacheManager;
    private final LocalOutlineCache local;

    public OutlinePublisher(
            OutlineRepository repository, CacheManager cacheManager, LocalOutlineCache local) {
        this.repository = repository;
        this.cacheManager = cacheManager;
        this.local = local;
    }

    public void publish(String tenantId, String courseId, LessonOutline outline) {
        repository.save(tenantId, courseId, outline);
        String key = LocalOutlineCache.cacheKey(tenantId, courseId);
        if (!TransactionSynchronizationManager.isSynchronizationActive()) {
            evict(key, tenantId, courseId);
            return;
        }
        TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() {
            @Override
            public void afterCommit() {
                evict(key, tenantId, courseId);
            }
        });
    }

    private void evict(String key, String tenantId, String courseId) {
        local.evict(tenantId, courseId);
        Cache shared = cacheManager.getCache("lesson-outline");
        if (shared != null) {
            shared.evict(key);
        }
    }
}

Cache.evict and CacheManager.getCache are Spring's cache SPI. getCache returns null when the name is unknown, so the publisher does not throw after a successful commit and leave the operator unsure whether the row landed. A missing cache is logged by the surrounding component; the TTL on any older entry still bounds the damage. repository.save runs in the caller's transaction. Registering the synchronization after the save, and only evicting afterCommit, is the race fix.

This publisher does not use @CacheEvict on the transactional method, on purpose, so the order is visible. @Cacheable(cacheNames = "lesson-outline", key = "#tenantId + ':' + #courseId") remains appropriate on a read method that is allowed to be a simple aside path when you are not implementing the lock yourself. Do not put both the hand-rolled lock and @Cacheable on the same method. Two caches will disagree about who fills the key.

HTTP headers for the two responses

The outline JSON from the application cache is still an authenticated response. Send Cache-Control: private, no-store so a shared CDN and an intermediate proxy do not store it. The application cache is server-side and is not an HTTP cache. Media URLs that embed a content hash send Cache-Control: public, max-age=31536000, immutable. Seat responses, if they exist at all, send private, no-store and are not placed in lesson-outline.

Decision record

ADR-106: Outlines are cacheable for a bounded window; seat counts are not cached

• Status: Accepted
• Context: Launch catalog QPS exceeds what the primary can serve on the serial outline query. A previous seat count inside a cached document oversold a cohort. Capture already recomputes price from the primary.
• Options: (A) No cache; scale the primary for a full miss. (B) Cache-aside outlines, TTL near 60 seconds plus jitter, evict after commit, local hot-key tier, no seats in the value, no-store on the HTTP response. (C) Write-behind the outline into the database.
• Decision: Option B. Redis outage fails outline reads rather than stampeding the primary at launch. Publish succeeds if the database commits, even when eviction fails. Seats are read and decremented only in the enrollment transaction.
• Consequences: A price can be stale on the outline page for up to about 70 seconds. Charging does not use that price. Operators must not add remainingSeats to LessonOutline. Write-behind is rejected because a lost cache is a lost publish.
• Reopen when: product requires the outline price to be current within a second, or a benchmark shows the fail-on-Redis-outage choice hurts more than a sized miss path.

Validation

• Publish a price change and read from a second node. The new price appears after commit without a manual flush, and a test that spies on the load sees no reload of the pre-commit row.
• Expire a hot key under concurrency in one process and observe a single loader call. Repeat with the lock held and observe that the losers do not all reach the repository.
• Put a seat field on the document in a test build and fail the build with an assertion that LessonOutline has no seat component. The type is the control.
• Run two enrollments against one remaining seat. Exactly one commits. Neither read a cache.
• Confirm the outline response's Cache-Control is private, no-store, and a hashed media URL is public and immutable.

Expected result

The launch reads outlines from caches. The primary sees fills and publishes, not the full burst. A price change is bounded. A seat change is exact. A Redis outage is an outline outage, not a database outage, and the record says that out loud.

Enterprise Real-World Scenarios

The price that lingered, and the charge that did not

A Tuesday publish moved a professional course from 199 to 249. The outline cache had a 24-hour TTL and no delete on publish, a combination someone defended with the hit ratio. The lesson page showed 199 until Wednesday. Checkout, rebuilt after the lesson 2 incident, recomputed the price from the primary and charged 249. Learners who had the tab open felt misled. Support could not say which number was "real," because both systems were working as designed and the designs disagreed.

The fix was not a longer explanation in the UI alone. ADR-106 cut the TTL to a minute, deleted the key after commit, and kept the capture recompute. The UI copy says the price is confirmed at payment. That sentence is true because the cache is not an input to the charge. Had capture used the cached outline, the company would have undercharged for a day and the cache would have become a financial system. The hit ratio stayed high at a one-minute TTL because the launch reads the same courses repeatedly. The long TTL had bought almost nothing and had cost a pricing argument.

A follow-up bug stored tenant A's negotiated price under a key of only the course id. Tenant B saw it for the rest of the TTL. The key change to include the tenant id shipped the same week, with a test that two tenants do not share an entry. Hit ratio dipped and was the wrong metric to page someone for. The right metric was "responses served from a key that does not contain the caller's tenant," which should stay at zero.

The last seat, sold twice

The compliance cohort capped at 500. The outline document included remainingSeats, filled by cache-aside with a 60-second TTL, because the page looked bare without it. At 09:00 several hundred learners enrolled in the same minute. The cache still said 40 seats were open. The enrollment path trusted it and skipped the conditional update "when the cache says there is room," a shortcut added to spare the primary. The shortcut sold past the cap. The primary, had it been asked, would have serialized the decrement.

The repair was structural. LessonOutline lost the field. Enrollment uses a conditional update on the seat row. The page stopped showing a live count during launch week and showed "limited seats" instead, which product accepted in the ADR. A later, uncached count endpoint was added for the instructor console, no-store, and it is allowed to be slow. The learner path does not call it in the hot window.

The stampede that accompanied the incident was independent and almost as expensive. The outline key expired at 09:00 because it had been filled at 08:59 by a warm-up job with no jitter. Every API node loaded the full outline at once, including the seat query that should not have been there. Jitter, the local loading cache, and the short lock made the next launch's primary graph flat while the hit ratio on the hot key stayed high. The seat bug would have survived a perfect stampede design. The stampede design would have survived a perfect seat rule. They are different failures, and the review wrote them as two notes so the next cache could not be justified by fixing only one.

Best Practices

• Write the staleness window in the scenario before you choose aside, through, or behind.
• Cache a document the reader can render, under one key that includes the tenant. Do not cache a join of independently updated fragments.
• Evict after commit. A TTL remains so a missed evict still ends.
• Jitter TTLs on keys that were filled together.
• Coalesce loads in-process with a loading cache. Across processes, use a lock that expires.
• Keep a small local tier for hot keys, with a TTL inside the window. Do not make the local tier the system of record.
• Disable caching of nulls on outlines. Do not cache exceptions as values.
• Leave seats, capture totals, and per-learner secrets out of shared entries. Prefer a type that cannot hold them.
• Send private, no-store on authenticated API bodies. Send long-lived public and immutable only on URLs that change when the bytes change.
• Page on staleness and on key shape, not only on hit ratio. A 24-hour hit ratio can be a bug.
• When the cache is down, choose stampede or fail-the-read explicitly and size the primary for the choice you made.
• Do not put @Cacheable and a hand-rolled loader on the same method.

Common Mistakes and Anti-Patterns

The cache as a source of truth. Seat counts or prices are read from the cache on the write path because the cache is "usually right." Under launch concurrency, usually is not the cap. The repair is a type that does not contain the fact, and a write transaction that touches only the primary.

Evict before commit. The miss path reloads the old row and resets the TTL, so publish extends staleness. It happens because @CacheEvict is easy to place on the method and the transaction boundary is invisible. The repair is afterCommit, with a test that a concurrent reader cannot store the pre-commit image.

One key, many tenants. The course id is unique inside a tenant and is used as a global key. Negotiated prices leak until the TTL. The repair is a key that includes the tenant, and a test with two tenants.

TTL as a hope, delete as a rumor. The window in the scenario is "immediate," the TTL is a day, and the delete is best-effort and unmonitored. The repair is to set the scenario to the TTL you can keep, then delete to beat it in the common case. If product rejects the TTL, do not cache that fact.

Stampede warm-up. A job fills every key at the same second with the same TTL. They expire together at the top of the hour. The repair is jitter, a loading cache, and a short lock. Warm-up is still useful; synchronized expiry is not.

Write-behind for catalog. The publish API returns success after a cache write, and a worker updates the database. A cache restart loses the course edit. The repair is to commit the database first. High-rate telemetry can use a durable log. An outline is not telemetry.

HTTP cache by accident. A CDN in front of the API stores GET /api/outlines/{id} because the path looks static. The next learner receives the previous learner's representation. The repair is private, no-store and a CDN configuration that does not cache the API host. Hit ratio at the edge is not a success metric for that path.

Lock without a timeout. A stampede lock is a key that only the holder deletes. The holder dies and the hot course 404s until an operator notices. The repair is setIfAbsent with a TTL of a few seconds, and a delete in finally for the happy path.

Troubleshooting and Design Review

Symptom: After publish, some nodes show the new price and others show the old price for about a TTL, then everyone converges.

Likely cause: the delete ran on one cache and not the local tiers, or the delete ran before commit and a miss refilled the old row, or a local TTL is long.

Investigation: the order of commit and evict in a trace. Compare local expireAfterWrite to the promise. Check whether afterCommit ran.

Corrective action: evict shared and local after commit. Keep the local TTL inside the window. Do not "fix" a split view by caching per node only; a deploy will make it worse.

Symptom: The primary's QPS spikes at the same minute every hour, and the hit ratio is otherwise excellent.

Likely cause: synchronized TTL expiry, often from a warm-up, on one or two hot courses.

Investigation: expiries by key, and the loader's call count per key per second. The aggregate hit ratio will look fine.

Corrective action: jitter, in-process coalescing, and the short lock. Confirm the next hour's primary graph before calling it done.

Symptom: Two tenants report each other's prices.

Likely cause: the cache key omitted the tenant, or an HTTP cache stored a response that varied by cookie and the cache ignored the cookie.

Investigation: a sample of keys, and Cache-Control on the response. Reproduce with two tenants in one test.

Corrective action: include the tenant in the server-side key, send private, no-store, and purge the CDN path if it stored anything. Rotate the cached documents; do not wait out a long TTL on a leak.

Symptom: Enrollment overshoots the cap only during the launch, and the outline payload contains a number of seats.

Likely cause: the write path trusts a cached count, or the count is display-only but a client submits it back and the server believes the client.

Investigation: whether the decrement's SQL is conditional. Whether the request body includes a seat count the server uses.

Corrective action: remove the field. Conditional update on the primary. Ignore any count the client sends. The outline cache is not part of the proof.

Symptom: When Redis restarts, the primary falls over, or the API returns errors for an hour.

Likely cause: the outage policy was implicit. Either every node stampeded, or a negative cache and a long TTL hid courses after Redis came back empty and then stored failures.

Investigation: database QPS during the restart, and whether exceptions were cached.

Corrective action: implement the ADR's choice on purpose. If the choice is fail-the-read, errors during the restart are expected and the primary should stay calm. If the choice is fall through, the primary must be sized for it. Never cache the exception.

Design-review questions: What is the source of this fact, and what age may a reader observe? Is the value safe for every caller who can form the key? Does eviction run after commit? What happens when the explicit delete fails? What happens when the hot key expires on every node at once? Is any field on the cached document used to authorize a write, a charge, or a seat? What does a cache outage do to the primary? Which HTTP directive stops a shared proxy from storing the response?

Interview Questions and Answers

Q1. What is cache-aside, and when is it the wrong pattern?

The application reads the cache, and on a miss reads the source and fills the cache. Writes update the source and then delete or replace the entry. It is right when the source must remain authoritative and a bounded stale read is acceptable, which is the outline page. It is wrong when the value cannot be stale at all, such as a seat count used to admit a learner, and it is wrong when you need the cache to be the only place that knows how to load data but you never installed a loader. If the cache dies, cache-aside still has the source, provided you have decided whether a miss storm is acceptable.

Q2. How do write-through and write-behind differ when the cache process crashes?

Write-through has already committed the source before success is returned, so a crash loses a cache entry and the next read reloads. The careful version deletes the key if the cache update fails, so an old entry cannot outlive the commit. Write-behind has acknowledged the write with the source still in the future, so a crash can lose the only copy. I will use write-behind only with a durability story I can name. I will not use it for a price or a seat. For outlines I do not even use write-through: cache-aside plus a short TTL meets the window without putting the cache on the publish latency path.

Q3. Why is evicting inside the transactional method not enough?

Because the eviction can become visible before the commit. A concurrent miss reads the old committed row and stores it under a new TTL, so the publish just made staleness worse. I register a transaction synchronization and evict in afterCommit. If the transaction rolls back I do not evict. The remaining gap, between the commit returning and the delete completing, is covered by the TTL the scenario already accepted.

Q4. What is a stampede, and how do you contain it without inventing a cache server feature?

A hot key misses and many callers load the source together. Inside one JVM, Caffeine's LoadingCache runs one loader and shares the result. Across API nodes I take a lock with ValueOperations.setIfAbsent and a TTL of a few seconds, let the winner fill the cache, and let losers read the filled value. The TTL means a dead winner does not hold the key. I also jitter the data TTL so keys filled together do not expire together. @Cacheable(sync = true) is a provider-specific hint; I do not assume it is a cluster-wide lock.

Q5. How do you handle a hot key if adding cache nodes does not move it?

A single key hashes to one slot, so more shards do not add capacity for that key. I put a small Caffeine cache on each API node, with a short expireAfterWrite inside the staleness window and refreshAfterWrite so an active node refreshes before expiry. The local tier is disposable on deploy. The shared tier is what a cold node reads. I do not pin learners to a node to keep a heap cache warm; lesson 26 already paid for that mistake.

Q6. What do you refuse to put in the outline cache, and why?

Remaining seats, the price a card will be charged, per-learner progress, and anything that belongs to another tenant. Seats change under concurrency exactly when the cache is most attractive, and a stale count oversells the cap. Capture recomputes money from the primary. Progress is private. The tenant id is part of the key so a negotiated price cannot leak. I encode the refusal in the type, not only in a wiki: LessonOutline has no seat field.

Q7. A cache outage happens during the launch. What should the API do?

I want an explicit choice. For this catalog, failing the outline read is better than sending the full burst to a primary we already know cannot take it. Publish still commits if the database is up; a missed eviction heals within the TTL. The other choice — fall through to the database — is valid only if the primary was sized for a 100 percent miss, which was option A in the ADR and we rejected it for launch. I will not decide this in the incident channel for the first time.

Q8. How do TTL and explicit invalidation work together?

The TTL is the promise you can keep when deletes fail. The delete is how you beat the promise in the common case. A design with only a TTL waits out the whole window after every publish. A design with only a delete has unbounded staleness when the delete errors. I set the scenario to the TTL, jitter it, and delete after commit. I monitor delete failures. I do not monitor only the hit ratio, which stays flattering while the price is a day old.

Q9. Why is a high hit ratio not an SLO?

Because a 24-hour TTL on a price produces a wonderful hit ratio and a false price. Because one hot key can be in pain while the aggregate is fine. Because a cache that stores the wrong tenant's document "hits." I measure freshness against publishes, the tenant prefix on keys, loader calls per hot key, and whether the write path touched the cache. Hit ratio is a capacity clue. It is not evidence the copy is allowed.

Q10. How do HTTP caches relate to this application cache?

They are another tier, operated by browsers and CDNs, with their own keys. private, no-store keeps authenticated outlines out of them. public, max-age=31536000, immutable is for media URLs that change when the bytes change, which is the edge path from lesson 26. An ETag can save bandwidth on a revalidation. It does not check enrollment. I do not put the API host behind a CDN policy aimed at static files and then try to repair it with a short max-age. The application cache already solved the launch QPS.

Q11. Cache-aside versus read-through in an interview design?

I say who runs the loader. If my service contains the miss branch, it is aside, and I must show the lock and the evict-after-commit. If the library contains the miss branch, it is read-through, and I must show that the loader is authorized and coalesced. Caffeine's LoadingCache is read-through inside one process. A shared Redis used only with get and set is not read-through; my code is aside. I do not claim the cache "handles" the database.

Q12. How does this lesson close the loop with capacity and with charging?

Lesson 24's launch envelope is a few thousand outline reads per second. Lesson 25 showed the primary cannot take that on the serial query. The cache exists so the primary sees fills, not the burst, and the ADR states the miss policy when the cache is gone. Lesson 2 said a cached price does not authorize a charge. ADR-106 repeats that as a consequence: staleness on the page is acceptable because capture does not read the page's copy. A design that uses the outline cache for both jobs has optimized the launch and corrupted the payment. I would rather fail a freshness interview question than blur those two reads.

Hands-On Exercises

Exercise: Caching for a product catalog and for inventory

Problem statement. A TechLearningPro storefront sells exam vouchers. The product page shows title, price, and description. Purchase decrements a voucher inventory. Design the caches. The launch burst is similar in shape to the catalog envelope in lesson 24; label any number you change.

Tasks.

1. Write two scenarios: product-page freshness after a price edit, and inventory correctness when two buyers take the last voucher. Give each a numeric measure.
2. Choose a pattern for the product document and a pattern for inventory. One of them may be "do not cache."
3. Define the cache key and the value. Show a tenant or currency dimension if the price can differ.
4. Specify TTL, jitter, and the moment of invalidation relative to commit.
5. Describe stampede control for the voucher that everyone opens at 09:00, inside a node and across nodes, using a real library behavior or setIfAbsent. Do not invent a server command.
6. State the cache-outage policy and what it does to the primary.
7. Set Cache-Control for the product page and for a hashed product image.
8. Write an ADR that rejects write-behind for inventory and rejects a cached count on the purchase path.

Expected deliverables. The two scenarios, the key and TTL design, the failure notes for stampede and outage, and the ADR.

Evaluation rubric.

A cached inventory count on the purchase path, a lock without a timeout, or a product key with no tenant or currency where prices differ means the design is not ready for review.

Lesson Summary

A cache is allowed to exist only where a reader may observe a stated age, a key that cannot cross tenants, and a source that remains authoritative. Cache-aside with a TTL and an after-commit delete fits TechLearningPro outlines. Read-through fits a loading cache inside one process. Write-through adds a synchronous update the publish path does not need. Write-behind risks the only copy of a price and is rejected. Stampedes are contained by coalesced loads, a lock with a TTL, and jitter. Hot keys are absorbed by a small local tier because more shards will not split one key.

Seat counts and capture totals stay on the primary. The outline page may be a minute stale. The charge and the cap may not. Authenticated responses are private, no-store. Immutable media URLs are the edge's problem, already sized in lesson 24. Hit ratio is not the goal. A bounded, tenant-safe, source-owned copy is the goal.

Lesson 28 turns to the database underneath these misses: indexes, the queries the loader actually runs, and the difference between a cache that hides a bad query and a query that should have been cheap on its own.

Further Reading

• RFC 9110 and RFC 9111 — HTTP semantics and HTTP caching, including Cache-Control. RFC 8246 — the immutable directive used for hashed media URLs.
• Spring Framework reference, "Cache Abstraction" — @Cacheable, @CacheEvict, @CachePut, and the sync attribute, including the warning that provider support varies.
• Caffeine wiki on population, refresh, and eviction — LoadingCache, expireAfterWrite, refreshAfterWrite, and coalesced loads inside one cache.
• Spring Data Redis reference for ValueOperations — get, set with a timeout, and setIfAbsent with a timeout, which this lesson uses as the lock. The class does not assemble command strings.
• Kleppmann, Designing Data-Intensive Applications, the chapter on derived data — why a copy has a freshness problem even when both systems are "up."
• Primers: /learn/high-level-design/cache-strategies, /learn/high-level-design/redis, /learn/system-design/caching-layers.
• This course: /learn/advanced-system-design/scalability-and-performance-fundamentals, /learn/advanced-system-design/load-balancing-and-traffic-management, /learn/advanced-system-design/database-design-and-indexing-for-scale, /learn/advanced-system-design/architectural-thinking-and-quality-attributes.