Rate Limiting, Throttling and Load Shedding

Advanced System Design & Software Architecture · Lesson 35 · High-Level Design and Distributed Systems

Lesson Overview

A cohort launch does two kinds of harm if the edge accepts every call. A single scripted client can open thousands of course-home requests and crowd out everyone else. Separately, a registrar upload and a popular homepage can together push catalog past the latency you promised, even though every caller is “legitimate.” The first problem is fairness: each learner and each tenant gets a budget. The second is survival: when the system is past its capacity, it rejects low-value work so high-value work still finishes. Those are rate limiting and load shedding. They are not the same control, and they do not return the same HTTP status for the same reason.

Rate limiting decides whether this caller is within a quota you set in advance: 20 course-home reads per second for a user, 1,000 per second for a tenant, 50 enrollment writes per minute. Excess is 429 Too Many Requests (RFC 6585) with Retry-After (RFC 9110). Throttling is the broader word for slowing or rejecting traffic to a target rate. A token bucket allows a burst and then a steady rate. A leaky bucket used as a shaper drains at a constant rate. A fixed window is easy to implement with Redis INCR and wrong at the boundary. A sliding window removes that boundary spike at a higher cost.

Load shedding is admission control for the process, not for the customer contract. When in-flight work or latency crosses a line, the server refuses new low-priority requests with 503 Service Unavailable and Retry-After. Health checks and an in-flight payment capture are not shed so that anonymous recommendation calls can keep their p99. Shedding with 429 blames the client for your capacity incident. Limiting with 503 tells a well-behaved learner they broke a rule they did not break.

The edge from API Gateway and Backend for Frontend is where coarse quotas run. Spring Cloud Gateway’s RequestRateLimiter with RedisRateLimiter is a token bucket (replenishRate, burstCapacity) keyed by a KeyResolver. Finer limits, and the shedding decision, live in the service that owns the scarce resource. This lesson implements a shared user-and-tenant token bucket with Redis commands and a Lua script executed through Spring Data Redis’s DefaultRedisScript, plus a local concurrency shedder. It does not invent a rate-limit SDK. When services later deploy independently, the same budgets are part of Microservices Architecture Principles, and the reserved concurrency for payments is the bulkhead in Circuit Breaker, Retry and Bulkhead Patterns.

Learning Objectives

By the end of this lesson, you should be able to:

1. Choose a token bucket, a leaky bucket, a fixed window, or a sliding window from the burst behavior you are willing to allow.
2. Enforce a per-user and a per-tenant quota together, so one tenant cannot win by creating more users.
3. Return 429 for quota exhaustion and 503 for load shedding, both with Retry-After, and explain the difference to a client author.
4. Implement an atomic token bucket in Redis with HMGET, HSET, and PEXPIRE inside a Lua script run by StringRedisTemplate.
5. Shed low-priority work first, and keep a reserved slice of concurrency for enrollment and payment capture.
6. Decide fail-open versus fail-closed when Redis is unreachable, per endpoint.
7. Place coarse limits at the gateway and resource-specific limits in the owning service.
8. State how these controls carry into the microservices module as per-service budgets and bulkheads.

Prerequisites

• Lesson 34: the gateway is the public choke point. This lesson puts a quota on that choke point and a second quota on the service.
• Lesson 26, Load Balancing and Traffic Management: shedding is useless if the load balancer keeps sending traffic to an instance that is already refusing work. Health checks must stay shallow.
• Lesson 24, Capacity Estimation and Workload Modeling: the numeric limits come from a capacity model, not from a round number in a code review.
• Lesson 27, Caching Strategies and Cache Invalidation: a cache absorbs repeated reads so the limiter is not your only defense. The limiter still exists for cache misses and for writes.
• Redis commands INCR, PEXPIRE, HSET, HMGET, and EVAL. Spring Data Redis StringRedisTemplate and DefaultRedisScript.

Detailed Core Concepts

Four algorithms, four burst stories

Fixed window. Count requests in a bucket named by the caller and the clock window, for example rl:user:42:202611061430 for one minute. Implementation: INCR the key and, if the new value is 1, PEXPIRE it for the window length. Do both in one Lua script. A lone INCR followed by EXPIRE from the client races: the process can die after the increment and leave a key that never expires, and two clients can both see a missing TTL. The limit is easy to explain: “100 per minute.” The defect is the boundary. Ninety requests at 12:00:59 and ninety at 12:01:00 are 180 requests in two seconds, all legal under a 100-per-minute fixed window. Login and coupon endpoints feel that spike.

Sliding window log. Keep every timestamp in a sorted set. ZADD the current time, ZREMRANGEBYSCORE drop entries older than the window, ZCARD count what remains. Accurate, and the memory is one entry per request. A user at 20 requests per second for a five-minute window is 6,000 members. Fine for a login route. Expensive as the only limiter for every catalog GET at 1,500 per second across many users.

Sliding window counter. Keep the previous window’s count and the current window’s count. Estimate the rate as previous * (1 - elapsed/window) + current. At 10 seconds into a 60-second window, the previous minute still contributes previous * 50/60. This smooths the boundary without storing every timestamp. It is an approximation: bursts inside a window can still pass if the weighted sum stays under the limit. Say that in the design so nobody claims exactness the math does not have.

Token bucket. The bucket holds up to capacity tokens. Tokens refill at r per second, and they stop at capacity. A request costs 1 token (or more, if a search is heavier than a health check). If the bucket has a token, take it and allow. If not, deny. A full bucket allows a burst of capacity, then the caller is held to r. That is the shape you want for a course-home API: a learner opening a screen may fire a handful of calls at once, and a script cannot sustain more than r. State is two numbers, tokens and the last refill timestamp, so Redis stores one hash per key.

Leaky bucket. Two definitions circulate, and reviews go wrong when they are mixed. As a queue (shaper), requests enter a buffer and a worker drains at a constant rate. Output is smooth. Excess waits or, if the buffer is full, is rejected. As a meter, the level rises with each request and leaks at a constant rate; a request that would raise the level above capacity is rejected. The meter’s steady state matches a token bucket’s sustained rate. The difference learners of the algorithm should be able to say out loud: a token bucket spends a saved burst immediately; a shaping leaky bucket never emits faster than the drain rate. TechLearningPro’s public read API uses a token bucket, because a small burst is good product behavior. A report exporter that must not stampede the warehouse uses a shaping queue with a fixed drain.

Policing rejects. Shaping delays. The HTTP API polices, because holding a connection to slow the client down ties up a thread. The export worker shapes, because the caller is a job, not a browser.

429, 503, and the headers you can actually standardize

RFC 6585 defines 429 Too Many Requests. RFC 9110 defines Retry-After as a delay in seconds or an HTTP date. Send delay-seconds. A date invites clock skew and a synchronized retry stampede. On a quota rejection:

httpCopy
HTTP/1.1 429 Too Many Requests
Retry-After: 2
RateLimit: "user";r=0;t=2

RateLimit in that form is the service-limit item in the IETF draft draft-ietf-httpapi-ratelimit-headers (r remaining, t seconds). It is a draft, not a finished RFC. The older split headers RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset appear in earlier drafts of the same work. X-RateLimit-* is common and inconsistent: some APIs put a timestamp in X-RateLimit-Reset, some put a delta. Document the one field you send. Do not send three dialects.

503 Service Unavailable with Retry-After is the shedding response. The client’s quota may be untouched. A client library that treats every 429 as “bug, stop forever” and every 503 as “retry immediately” will do the wrong thing on both. Publish one paragraph in the API docs: 429 means slow down to the quoted limit; 503 means the platform is protecting itself, honor Retry-After, and try again with jitter.

Do not use 400 for a quota. Do not use 401; the caller is authenticated and still over budget. Do not use 200 with an error body. Intermediaries and client generators understand status codes.

Keys, fairness, and who the limit is about

A limit without a key is a global fuse. It trips for everyone because one tenant looped. Keys TechLearningPro uses:

A request must pass every applicable bucket. The user is under their own limit and the tenant is under the tenant limit. Checking only the user lets a tenant with 50,000 accounts overwhelm catalog. Checking only the tenant lets one user inside a quiet tenant run a scraper. Anonymous routes have no sub. Limit them by IP and by a coarse global ceiling, and do not pretend the IP is a person. Carrier-grade NAT shares an address across many learners; an IP limit on authenticated traffic punishes a campus. Use IP limits on login, where the alternative is an open credential-stuffing port.

The gateway’s KeyResolver should use the JWT subject for learner routes and the client id for the partner host. Spring Cloud Gateway’s RedisRateLimiter is one token bucket per resolved key. A single gateway filter does not express “user and tenant” unless you install two limiters or you do the check in the BFF. This lesson puts the hierarchical check in a servlet filter on the BFF and the service, and leaves the gateway limiter as a coarse per-principal fuse so a broken client never reaches the BFF at full line rate.

Load shedding is a priority list

Capacity is in-flight work and latency, not a feeling. Each instance tracks concurrent requests. Budgets from the capacity model, for one instance that handles about 200 in-flight course reads before p99 breaks:

Shedding P4 and then P3 keeps the instance inside its latency SLO for P1 and P2. A global semaphore that treats every request alike lets recommendations occupy every thread while enrollment times out. That is the bulkhead, applied as admission rather than as a thread pool name. Lesson 41 implements the pool. This lesson decides which new request is refused.

Do not shed a request you have already started charging for. The capture callback is P0 because dropping it loses the link between the acquirer and your ledger; the repair is a reconciliation job, not a cleaner p99. Health checks stay local and cheap. If health depends on Redis, a Redis blip drains the fleet at the load balancer and turns a limiter failure into an outage. Lesson 34’s shallow health check is a shedding requirement.

The signal to start shedding can be static (in-flight above N) or adaptive (p99 above the SLO for a minute). Static is explainable and reviewable. Adaptive controllers, including the Vegas-style limit in Netflix’s concurrency-limits library, move the limit as latency moves. They need damping or they oscillate. Start static. Add adaptive only with a chart that shows the limit and the latency on the same board.

When the counter store is down

Redis is a dependency of the limiter. Choose the failure mode per route:

• Login and coupon: fail closed, or fall back to a much tighter in-process limit. An open login during a Redis outage is a credential-stuffing window.
• Course home: fail open onto a local token bucket per instance, sized to that instance’s share of capacity. Availability of the product beats a perfect global count. The local bucket will not coordinate across instances, so the real admission is a bit higher than the global quota until Redis returns. That is the documented degradation.
• Enrollment write: fail closed if you cannot enforce the write quota, or accept only if the local limit says the instance is idle. A duplicate enrollment is worse than a 503 the client will retry with an idempotency key.

Log a metric rate_limit_backend_unavailable so fail-open is visible. A silent fail-open becomes the permanent architecture the day Redis is undersized and nobody notices the limiter stopped.

Architecture and Design Diagrams

mermaidCopy
flowchart TD
  Req[Request] --> Auth[JWT already validated]
  Auth --> GW[Gateway coarse token bucket]
  GW -->|over quota| R429[429 Retry-After]
  GW -->|allowed| Shed{Instance over budget?}
  Shed -->|P4 or P3 and saturated| R503[503 Retry-After]
  Shed -->|admitted| Hier[User bucket and tenant bucket]
  Hier -->|either denies| R429
  Hier -->|both allow| App[Catalog or enrollment handler]

mermaidCopy
flowchart LR
  subgraph redis [Redis hash per key]
    T[tokens]
    TS[ts millis]
  end
  F[OncePerRequestFilter] -->|EVAL Lua| redis
  F -->|allowed| Chain[Controller]

mermaidCopy
sequenceDiagram
  participant C as Client
  participant G as Gateway
  participant B as web-bff
  participant R as Redis
  C->>G: GET /web/courses/42
  G->>G: coarse principal bucket
  G->>B: forward
  B->>B: shed if P4 and in-flight high
  B->>R: token bucket user and tenant
  R-->>B: tenant denied
  B-->>C: 429 Retry-After 1

Trust boundary: the limiter key comes from the validated JWT sub and a tenant claim the identity provider put in the token, not from a client header. Redis is reachable only from the application network. The Lua script’s keys are your keys. A caller who could EVAL arbitrary scripts against this Redis could bypass the limit; application credentials for Redis do not include administrative commands in production ACLs (ACL SETUSER restricting the app to the rl: prefix is a Redis ACL, configured on the server, not a client invention).

Technical Explanation

Integer token bucket

Store milli-tokens so the script stays in integers. Capacity 40 tokens is 40,000 milli-tokens. A refill of 20 tokens per second is 20 milli-tokens per millisecond.

luaCopy
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_per_ms = tonumber(ARGV[2])
local now_ms = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local ttl_ms = tonumber(ARGV[5])
local data = redis.call('HMGET', key, 'tokens', 'ts')
local tokens = tonumber(data[1])
local ts = tonumber(data[2])
if tokens == nil then
  tokens = capacity
  ts = now_ms
end
local elapsed = math.max(0, now_ms - ts)
tokens = math.min(capacity, tokens + elapsed * refill_per_ms)
local allowed = 0
if tokens >= cost then
  tokens = tokens - cost
  allowed = 1
end
redis.call('HSET', key, 'tokens', tokens, 'ts', now_ms)
redis.call('PEXPIRE', key, ttl_ms)
return allowed

EVAL runs the script atomically on the Redis server. Two app instances cannot both read 1 token and both proceed. PEXPIRE covers idle keys so a learner who visited once does not leave a hash forever. Set the TTL to several windows, not to one refill interval, or a bucket disappears mid-burst and refills to full, which grants a free burst. Clock source is the application’s System.currentTimeMillis() passed in as ARGV. Redis is not asked for the time inside the script, so you do not depend on TIME and on app clocks agreeing beyond “good enough for a limiter.” A multi-second skew between app nodes makes buckets slightly more generous. It does not reset them. If you need one clock, read Redis TIME inside the script instead of passing now_ms. Pick one and document it.

Spring Data Redis:

javaCopy
final class TokenBucket {
  private final StringRedisTemplate redis;
  private final DefaultRedisScript<Long> script;

  TokenBucket(StringRedisTemplate redis) {
    this.redis = redis;
    this.script = new DefaultRedisScript<>();
    this.script.setResultType(Long.class);
    this.script.setScriptText(TOKEN_BUCKET_LUA);
  }

  boolean allow(String key, long capacityMilli, long refillPerMs, long costMilli) {
    Long allowed = redis.execute(
        script,
        List.of(key),
        Long.toString(capacityMilli),
        Long.toString(refillPerMs),
        Long.toString(System.currentTimeMillis()),
        Long.toString(costMilli),
        "120000");
    return Long.valueOf(1L).equals(allowed);
  }
}

DefaultRedisScript and StringRedisTemplate.execute are Spring Data Redis. The script returns a number; Long matches. A null return means the call did not run (connection failure). Treat null as backend unavailable and apply the fail-open or fail-closed policy. Do not treat null as “allowed” by accident in a boolean conversion that maps null to false on the login route and to true on the home route without a log line.

Check user and tenant in two allow calls. If the user bucket consumes a token and the tenant bucket then denies, you have charged the user for a rejected request. For a limiter that is annoying and not fatal: the user is slightly under their budget. If you need both-or-neither, run one script with two keys. Redis Cluster requires both keys in the same hash slot. Use a hash tag: rl:{tenantId}:user:{sub} and rl:{tenantId}:tenant, so {tenantId} fixes the slot. Without a hash tag, a multi-key script errors on a cluster. A single-node Redis hides that bug until you scale out.

Fixed window, for contrast

luaCopy
local n = redis.call('INCR', KEYS[1])
if n == 1 then
  redis.call('PEXPIRE', KEYS[1], ARGV[1])
end
return n

Allow when n <= limit. This is the script to show in review when someone proposes INCR plus EXPIRE from Java. Use it for a coarse daily quota (“500 enrollments per tenant per day”) where a boundary spike of two days’ worth in two seconds is unacceptable — so do not use it there. Use it when the product rule truly is “this calendar minute,” and pair it with a token bucket if abuse at the boundary matters.

Gateway token bucket

Spring Cloud Gateway’s built-in limiter is a token bucket in Redis. One route argument set:

yamlCopy
filters:
  - name: RequestRateLimiter
    args:
      redis-rate-limiter.replenishRate: 20
      redis-rate-limiter.burstCapacity: 40
      redis-rate-limiter.requestedTokens: 1

replenishRate is tokens per second. burstCapacity is the bucket size. requestedTokens is the cost of this request. The key comes from a KeyResolver bean. Resolve the authenticated principal:

javaCopy
@Bean
public KeyResolver principalKeyResolver() {
  return exchange -> exchange.getPrincipal()
      .map(Principal::getName)
      .defaultIfEmpty("anonymous");
}

getPrincipal() is empty if this filter runs before authentication. Order the gateway filters so the JWT is authenticated first. An “anonymous” bucket shared by every failed lookup is a single global limit that trips for the whole internet. Fall back to the peer address only on routes that are supposed to be anonymous, and name that resolver so it cannot be reused on /web/**.

This gateway limiter is the coarse fuse. It does not know tenant-plus-user. The BFF filter does.

Shedding in the service

javaCopy
if (priority == Priority.RECOMMENDATION && inFlight.get() >= softLimit) {
  response.setStatus(HttpStatus.SERVICE_UNAVAILABLE.value());
  response.setHeader(HttpHeaders.RETRY_AFTER, "1");
  return;
}
if (priority == Priority.ENROLLMENT && enrollmentInFlight.get() >= enrollmentLimit) {
  response.setStatus(HttpStatus.SERVICE_UNAVAILABLE.value());
  response.setHeader(HttpHeaders.RETRY_AFTER, "2");
  return;
}
inFlight.incrementAndGet();
try {
  chain.doFilter(request, response);
} finally {
  inFlight.decrementAndGet();
}

softLimit is below the thread pool’s hard maximum so the process can still accept P0 and P1. enrollmentInFlight is the reserved slice. Recommendations do not increment that counter. A capture callback uses neither shedder. The numbers come from a load test that finds the knee of p99, then backs off. A limit copied from a blog will shed too late, after the threads are already stuck, or too early, during ordinary evening traffic.

Put jitter in any client retry of 503. Retry-After: 1 plus a random few hundred milliseconds avoids a synchronized herd. The server cannot force the client to comply. Document it, and make the server safe if clients ignore it: the shedder is still there on the next wave.

Practical Implementation or Design Walkthrough

Problem

Protect TechLearningPro’s course-home read and enrollment write. Evening peak is 1,500 home reads per second across all learners. One tenant may be a university of 20,000 students. A single account must not sustain more than 20 reads per second. A tenant must not sustain more than 1,000. Enrollment is 5 per user per minute. When an instance is past 150 in-flight home reads, recommendation traffic is shed, then anonymous browse, before authenticated home reads are refused.

Requirements

• 429 when a user or tenant bucket is empty, with Retry-After in seconds.
• 503 when the instance sheds, including for a caller who is inside their quota.
• Atomic updates in Redis. No check-then-act from Java.
• Login fails closed if Redis is down. Course home falls back to a per-instance bucket and emits a metric.
• Gateway fuse at the principal, service-level hierarchical buckets behind it.
• Shallow health checks, excluded from shedding and from the quota.

Design

Home read costs 1,000 milli-tokens. User bucket: capacity 40,000 (burst 40), refill 20 per second. Tenant bucket: capacity 2,000,000, refill 1,000 per second. Keys rl:{tid}:user:{sub}:read and rl:{tid}:tenant:read, hash-tagged for cluster safety. Enrollment uses a separate pair of keys and a smaller refill so a read budget cannot be spent as writes.

The filter order on the BFF is: authentication, shedding, rate limit, controller. Shedding before the Redis call means an overloaded instance does not spend Redis round trips rejecting recommendations. Quota before the controller means a denied enrollment does not open a database transaction.

Enrollment’s idempotency key (lesson 37) is still required. A 429 means the write did not happen. A client retry with the same key after Retry-After is safe. A 503 on a write that already passed the limiter and then died inside the handler is the idempotency case, not the limiter case. Do not conflate them in the client: 429, wait, retry the same key; 503, wait, retry the same key; 200, store the body.

Failure handling

Retry-After for a token bucket is ceil(cost / refillRate) seconds, at least 1. Do not return 0. A client that retries immediately will stampede.

Tests

• Two threads, one token: one allow and one deny, using Testcontainers Redis or a Redis the build already runs. An embedded fake that ignores atomicity does not test the script.
• Fixed-window boundary demonstration: 100 at the end of a window and 100 at the start, as a unit test of the fixed script, so the team sees the spike and does not “simplify” the home route back to INCR.
• Filter test: recommendation receives 503 when inFlight is at the soft limit; enrollment still proceeds while under enrollmentLimit.
• Redis stopped: home filter records the fallback metric and uses the local bucket; login filter returns 503.
• A request with a client-supplied X-User-Id and no JWT does not consume another user’s bucket.

Enterprise Real-World Scenarios

Scenario 1: Evening course home and one noisy tenant

Business context. 1,500 reads per second globally is within catalog’s budget. One tenant’s integration bug sends 800 per second from a few service accounts. Other tenants’ learners see catalog latency climb.

Choices. The tenant bucket at 1,000 per second would still allow this bug to take most of the region, so the production number is set from the capacity model: no single tenant more than 30 percent of regional read capacity. If regional capacity is 1,500, the tenant ceiling is 450, not 1,000. The walkthrough’s 1,000 was the product starting point; the review changes it when the math says so. The user buckets stop a single account. The gateway fuse at 20 per second per principal catches the bug before the BFF. Authenticated learners in other tenants stay on P2 and are not shed. The noisy tenant receives 429 and a Retry-After. Support has a chart of denials by tenant id.

What fails review. A global INCR for the whole region. One bug becomes everyone’s 429. An IP limit on authenticated traffic. A campus NAT turns a lecture hall into a self-inflicted outage.

Scenario 2: Launch enrollment versus recommendation traffic

Business context. Enrollment opens at 09:00. Recommendations on the same instances keep their normal QPS. Threads fill. p99 on enrollment exceeds the SLO while recommendation responses are still 200.

Choices. Recommendations are P4 and shed at the soft in-flight limit with 503. Anonymous browse sheds next. Enrollment uses a reserved counter and is refused with 503 only when that slice is full, which is a capacity incident you page on, not a fairness issue. Payment capture callbacks are excluded. The limiter on enrollment writes still returns 429 for a user retrying faster than 5 per minute, so a client bug does not consume the reserved slice. Clients retry 503 and 429 with the same idempotency key and with jitter. Lesson 41’s bulkhead is the next step if recommendations and enrollment should not even share a process. Until that split, admission priority is what keeps the launch inside its latency budget.

The microservices bridge: each service publishes its own limit and shed thresholds. The gateway does not know catalog’s thread count. Observability: Logging, Metrics and Tracing is where denial counts, shed counts, and Redis errors become dashboards. A limit you cannot see will be tuned by outage.

Best Practices

• Pick the algorithm from the burst you intend to allow. Token bucket for interactive reads. Shaping queue for batch exports. Fixed window only when the product language is a calendar bucket and the boundary spike is harmless.
• Always combine user and tenant (or partner client) limits on shared infrastructure. One key is not fairness.
• Return 429 for quota and 503 for shedding. Send Retry-After in seconds. Document the draft RateLimit field if you send it, and do not also send a contradictory X-RateLimit-Reset.
• Make the Redis update atomic with Lua. Hash-tag keys that a single script touches on a cluster.
• Take identity from the validated token. Ignore client-supplied user ids.
• Fail closed on abuse-sensitive routes. Fail open to a local bucket on reads, and metric it.
• Shed in priority order. Reserve concurrency for payments and enrollment. Keep health checks out of both mechanisms.
• Set numbers from a load test and a capacity model. Revisit them when the region’s capacity changes.
• Put a coarse fuse at the gateway and the hierarchical limit next to the resource.
• Add jitter to client retries. Assume some clients will ignore Retry-After, and stay safe when they do.

Common Mistakes and Anti-Patterns

Fixed window on login. The boundary doubles the allowed rate for a second, which is exactly what a credential-stuffing burst exploits. Use a token bucket or a sliding window, and fail closed if the store is down.

Check in Java, increment in Redis. Two instances read “under the limit” and both allow. The quota is a suggestion. The script that decides and updates together is the implementation. A GET followed by SET is the bug.

429 for a capacity incident. Learners who sent one request get “too many requests” during your partial outage. Client teams add aggressive retries because the status seems unfair. Shed with 503, and keep 429 for the quota they can look up in your docs.

One shared anonymous bucket. The KeyResolver returns "anonymous" whenever the principal is missing, including when the filter order is wrong. The first thousand real users share a single budget and receive 429. Fix the order, and give anonymous routes an explicit IP resolver.

Shedding health checks. The load balancer marks every instance down. The region goes dark while the process was only trying to refuse recommendations. Exclude /actuator/health.

Failing open on login with no metric. Redis maintenance becomes an open credential endpoint, and the postmortem cannot see it. Fail closed, or fall back to a tiny local limit and page on the fallback.

Limits only at the gateway. The gateway counts calls. It does not know that one search costs fifty catalog reads, or that enrollment is the scarce write. The owning service limits the expensive operation. Gateway limits stop a broken client from connecting at line rate.

Retry-After of zero, or no jitter. Every client retries on the same millisecond. The shedder oscillates. Send at least one second, and jitter in the client.

Troubleshooting and Design Review

Design review:

• What is the burst a real client needs, and what sustained rate is abuse?
• Which keys must all pass?
• Where does identity come from?
• What is the status code for quota versus overload?
• What happens when Redis is unreachable, per route?
• What is shed first, and what is never shed?
• How does the client retry, and which key does it reuse?
• Which chart shows denials by key before the next launch?

Interview Questions and Answers

Q1. What is the difference between a token bucket and a fixed window?

A fixed window counts requests in a clock bucket and resets the count when the window rolls. It is a single INCR. At the boundary a caller can spend nearly two windows of budget in a few seconds. A token bucket refills continuously up to a capacity. The caller may spend a burst equal to the capacity, then only the refill rate. Interactive course-home traffic wants that burst. Login protection does not want the fixed-window cliff. If the product text says “100 per calendar minute” and the boundary spike is acceptable, the fixed window matches the sentence. Say which one you implemented.

Q2. How do you keep the Redis update atomic?

Put the read, the refill math, the decrement, and the PEXPIRE in one Lua script and run it with EVAL via DefaultRedisScript on StringRedisTemplate. Redis runs the script without interleaving other commands. A Java sequence of HMGET then HSET allows two instances to allow the last token. On Redis Cluster, keys touched by one script must share a hash slot, so include a {tenantId} hash tag. Treat a null return from execute as “Redis was not reached,” not as a numeric deny.

Q3. Why do you limit by user and by tenant?

Per-user limits stop one account. A tenant with thousands of accounts stays under every user limit and still overwhelms the region. Per-tenant limits stop that, and they do nothing to one noisy user inside a large unused budget. Require both. Partner client ids are a third key space so a registrar upload does not spend the learner tenant’s budget. Take the ids from validated claims.

Q4. When do you return 429, and when 503?

429 means this caller exceeded a quota you documented. RFC 6585 defines that status. 503 means the server is shedding load to stay within capacity; the caller may be well under quota. Both should carry Retry-After as delay-seconds (RFC 9110). Using 429 for shedding trains clients to treat overload as their bug. Using 503 for a quota hides the fairness rule. Health checks are neither: they return 200 while the instance can serve, and they are not counted against a user’s bucket.

Q5. What is load shedding, in priority order?

Admission control when in-flight work or latency crosses a threshold. Refuse the cheapest, least critical new work first: recommendations, then anonymous browse, then authenticated reads, and only then new enrollments. Do not refuse health checks or payment callbacks you must reconcile. A single semaphore for all traffic lets low-value calls occupy the threads high-value calls need. Reserved counters are the bulkhead at the door. Lesson 41 names the same idea inside the process.

Q6. How should the limiter behave when Redis is down?

It depends on the route. Login and other abuse surfaces fail closed or fall back to a very small local limit, and they page. Course-home reads fail open onto a per-instance token bucket so a Redis incident is not a total outage, and they emit a metric so the degradation is visible. The local bucket does not coordinate globally; the quota is temporarily looser. Health checks do not call Redis. Write this policy next to the route. A single boolean failOpen=true for the whole process is how login becomes unprotected.

Q7. Where does the gateway limiter stop, and where does the service limiter start?

The gateway’s RequestRateLimiter and RedisRateLimiter are a coarse token bucket per KeyResolver result, with replenishRate and burstCapacity. They stop a broken client before it occupies BFF threads. They do not express “user and tenant,” and they do not know that a search is fifty times a home read. The service that pays the cost enforces the hierarchical buckets and the shedder. Two layers with the same key and the same number are just latency. Two layers with different jobs are the design.

Q8. Why is a sliding-window log accurate and often unused?

It stores each timestamp and counts those inside the window, so the boundary spike of a fixed window is gone and the estimate of a sliding counter is unnecessary. Memory and write rate grow with request rate times window length. At home-read volume that set is large. Use the log for low-rate sensitive routes. Use a token bucket, two numbers per key, for the high-rate read path. The sliding counter is the compromise when you must say “per minute” and you can tolerate an approximation.

Q9. A client retries a 503 enrollment with a new idempotency key. What breaks?

The shedder refused the call, or the call died after the handler started. A new key looks like a new enrollment. If the first attempt actually captured a seat, the second creates another. The client should send the same idempotency key on 429 and on 503. The server’s enrollment handler, not the limiter, deduplicates that key. The limiter’s job is only to decide whether this attempt may enter. Lesson 37 specifies the stored response. Quote both behaviors in the client guide so the retry policy is not invented per app.

Q10. How do you pick the numeric limits?

From a load test that finds the knee where p99 breaks, and from the capacity model’s regional budget. A tenant ceiling is a fraction of that budget so one tenant cannot consume the region. A user ceiling is a burst a real screen needs, above normal use and below a scraper. Revisit the numbers when you add instances: a per-instance shedder limit scales with the instance, while the Redis quota is global and does not automatically grow. Shipping a global quota tuned for one node, then running ten nodes, under-uses the fleet. Shipping a per-node quota of the full regional budget, times ten nodes, over-admits. Write which number is global and which is per instance.

Hands-On Exercises

Exercise 1: User, tenant, and shedder

Problem. Specify limits for TechLearningPro course-home reads and enrollment writes at 1,500 reads per second regional peak, with one tenant capped at a fair share.

Tasks.

1. Choose an algorithm for reads and for enrollment. State the burst it allows.
2. Define Redis keys for user and tenant, including a hash tag.
3. Specify 429 and 503 cases, with Retry-After.
4. Define shed order and what is excluded.
5. Define Redis-down behavior for login, home, and enrollment.
6. Say what the gateway limits versus the BFF.
7. State how an enrollment retry identifies itself.

Expected deliverables. Key table, status table, Lua or a precise description of the atomic update, shed thresholds, failure policy.

Acceptance. A tenant cannot exceed its share by adding users. A boundary spike is either accepted explicitly or designed out. Redis loss does not open login and does not take health checks down. Enrollment retries do not require a new idempotency key.

Exercise 2: Review an INCR filter

The filter increments rl:global with INCR, expires it every minute from Java, returns 429 when the count exceeds 100,000, and uses the same path for health checks. Write the review: atomicity, fairness, status code, health, and the algorithm’s boundary.

Rubric.

Lesson Summary

Rate limits allocate a budget you decided in advance. Token buckets allow a bounded burst and then a steady rate; fixed windows are simpler and spike at the boundary; sliding logs are exact and heavier; a shaping leaky bucket smooths output for batch work. Enforce the user and the tenant together, atomically, in a Redis Lua script, with keys taken from the validated token. Return 429 when the caller is over quota and 503 when the instance sheds low-priority work, both with Retry-After. Do not shed health or payment callbacks. Fail closed on login and fail open to a local bucket on reads if Redis is down. The gateway carries a coarse fuse. The service carries the hierarchical limit and the priority shedder. Those same numbers become per-service contracts in the microservices module, with bulkheads and dashboards so a launch spends capacity on enrollment rather than on recommendations.

Further Reading

• RFC 6585, which registers 429 Too Many Requests, and RFC 9110, section on Retry-After.
• IETF draft RateLimit header fields for HTTP (draft-ietf-httpapi-ratelimit-headers). Still a draft: RateLimit: "default";r=50;t=30. Earlier revisions discussed RateLimit-Limit and RateLimit-Remaining. X-RateLimit-* is not that document.
• Redis documentation on EVAL and Lua and hash tags for Cluster.
• Spring Data Redis, RedisTemplate / StringRedisTemplate and DefaultRedisScript in the Spring Data Redis reference.
• Spring Cloud Gateway, RequestRateLimiter — RedisRateLimiter, replenishRate, burstCapacity, KeyResolver.
• Next: Distributed Locking and Coordination, Idempotency and Exactly-Once Processing, and Module 4 Circuit Breaker, Retry and Bulkhead Patterns, Observability: Logging, Metrics and Tracing.