Distributed Locking and Coordination

Advanced System Design & Software Architecture · Lesson 36 · High-Level Design and Distributed Systems

Lesson Overview

Two TechLearningPro workers wake up at 02:00 and both try to generate the October invoice export. The correct outcome is one export file. A unique row on (period) in PostgreSQL produces that outcome with no lock service: the second insert fails, and the winner proceeds. A Redis key with a 30-second TTL produces that outcome only while the winner finishes inside 30 seconds, the process does not pause, the key is not lost on failover, and the loser does not start because it failed to read the key. Those conditions are a lease, not a proof that two writers cannot exist.

This lesson is about when a lock is the wrong tool, and how to use one when a database constraint cannot express the rule. Mutual exclusion means one holder performs the work. A lease is exclusion that expires, which every lock built on a timeout actually is. A fencing token is a number that increases each time the lease is granted, which the protected store checks so a paused former holder cannot write after a new holder exists. Without that check, the lock is an optimization that reduces how often two workers run. It is not a correctness mechanism.

Redis’s documented lock is SET key randomValue NX PX ttl, released by a Lua script that deletes the key only if the value still matches. It does not issue a fencing token. Redlock, which tries the same idea on several independent Redis nodes, is not a consensus algorithm; Martin Kleppmann’s critique is part of the design discussion below, and this course does not use Redlock to protect invoices or seat assignment. ZooKeeper gives you sessions and ephemeral nodes, and Apache Curator’s InterProcessMutex implements the lock recipe on top. The token you can fence with is still something the database must verify. PostgreSQL can often do the whole job: a unique constraint, INSERT ... ON CONFLICT, SELECT ... FOR UPDATE SKIP LOCKED, or pg_try_advisory_xact_lock.

Coordination beyond a single lock — who is the scheduler leader, which Kafka consumer owns a partition — belongs to a consensus system or to the component that already runs an election. You do not build that from SET NX. Module 4’s services will be tempted to “just take a lock” around a cross-service workflow. Microservices Architecture Principles and Idempotency and Exactly-Once Processing are the alternatives: make the write safe to repeat, and let the database be the source of exclusion.

Learning Objectives

By the end of this lesson, you should be able to:

1. Prefer a unique constraint or SKIP LOCKED when the resource lives in a database you already update.
2. Explain a Redis lock as a lease: SET NX PX, value-checked unlock, expiry while the holder is paused.
3. State why Redlock is not the correctness story for payments or inventory, and what Kleppmann’s fencing-token argument requires the storage system to do.
4. Use a monotonic fencing token on every write so a stale holder affects zero rows.
5. Describe a ZooKeeper ephemeral node and a Curator InterProcessMutex as a session-bound lease, including session timeout versus GC pauses.
6. Choose a leader only when one process must run a singleton task, and store that leadership where a crash releases it.
7. List the crash points of a job worker: death before completion, death after completion before acknowledgement, lock expiry during a long job.
8. Carry the rule into microservices: do not lock across services; lock or constrain inside the service that owns the row.

Prerequisites

• Lesson 28, Database Design and Indexing for Scale: unique indexes are the exclusion primitive this lesson reaches for first.
• Lesson 31, CAP Theorem and Consistency Models: a lock service has its own partition behavior. A lock that “fails open” when the lock service is unreachable is two holders.
• Lesson 33: a consumer group already assigns each partition to one member. That is coordination for the log. It is not a general mutex for an invoice job.
• Lesson 35’s Redis use is quotas. A rate-limit key and a lock key solve different problems. Do not overload one Redis key as both.
• SQL transactions, and the idea that a client pause (garbage collection, a long HTTP call, a debugger) is longer than a timeout you configured.

Detailed Core Concepts

Exclusion without a lock service

The invoice export’s invariant is “one row per period.” PostgreSQL enforces it:

sqlCopy
create table billing.invoice_export (
  period text primary key,
  status text not null,
  fencing_token bigint not null,
  started_at timestamptz not null
);

The worker inserts ('2026-10', 'RUNNING', 1, now()). The loser gets a unique violation and stops. There is no TTL, no second system, no unlock bug. If the winner dies, the row stays RUNNING. A sweeper may reclaim a row whose started_at is older than the worst-case runtime only by advancing a token and checking it on write, or by requiring the export to be idempotent so a second run overwrites the same object name. The stuck row is an operational problem you can query. A vanished Redis key is a problem you discover when two files exist.

Use this pattern when the thing you are protecting is already a row: one export per period, one seat per (offering, learner), one payment per idempotency key (lesson 37). The unique index is the lock, and it does not expire while the holder is in a garbage-collection pause.

INSERT ... ON CONFLICT DO NOTHING RETURNING id makes the “did I win” check a single statement. Zero rows returned means someone else owns the period.

Job queues and SKIP LOCKED

When many jobs are ready and any worker may take the next one, a unique key on the job id does not assign the job. PostgreSQL’s queue pattern does:

sqlCopy
select id, fencing_token
from jobs.task
where status = 'READY' and run_at <= now()
order by run_at
for update skip locked
limit 1;

FOR UPDATE locks the selected row for this transaction. SKIP LOCKED skips rows another transaction already locked, instead of waiting. The worker then updates the row to RUNNING, increments fencing_token, sets locked_until, and commits. Other workers skip it. This is the right design for TechLearningPro’s certificate-generation queue. It uses the database you already run. It does not need Redis to be correct.

The crash window is the work after commit. The worker committed RUNNING and then died during PDF generation. The row is not READY, so nobody else selects it. You need locked_until. A reaper selects RUNNING rows whose lease expired, increments fencing_token, and sets them READY again. The original worker, if it is still alive and finishes, writes WHERE id = ? AND fencing_token = ? and updates zero rows. It deletes its PDF attempt or overwrites nothing. That WHERE clause is the fence.

Advisory locks (pg_try_advisory_xact_lock(key)) are a session or transaction mutex inside one Postgres cluster, useful when there is no natural row. pg_try_advisory_xact_lock releases at transaction end. They do not coordinate a worker that must hold exclusion across several transactions while it calls S3. For that, the lease column on the row is the honest tool, because it survives the connection returning to the pool. An advisory lock held on a pooled connection is a classic bug: the pool gives that connection to another request, which inherits the lock, or the pool’s “idle connection” still holds it.

The Redis lease, stated exactly

The Redis documentation’s single-instance lock is:

textCopy
SET lock:invoice:2026-10 <random> NX PX 30000

NX sets the key only if it is absent. PX 30000 sets a 30-second millisecond TTL. The value is a random token this holder generated (a UUID). The holder releases the lock with a script, not with DEL:

luaCopy
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
else
  return 0
end

DEL without the compare is how holder A’s slow unlock deletes holder B’s lock after A’s TTL expired and B acquired. Spring Data Redis maps the acquire to StringRedisTemplate.opsForValue().setIfAbsent(key, token, Duration.ofMillis(30_000)). Run the unlock with DefaultRedisScript<Long>, the same mechanism as lesson 35. Both calls are real library methods. Neither call is a fencing token.

What this lease guarantees when Redis is one primary and the TTL has not fired: at most one client that obeys the protocol thinks it holds the lock. What it does not guarantee:

• The holder finishes before the TTL. A GC pause, a long PDF render, or a stalled disk can outlast 30 seconds. The key disappears. Another worker acquires. The first worker resumes. Two exporters run.
• A failover to a replica that never received the key. Depending on replication and persistence, the lock vanishes and a second holder appears while the first still runs. Redis is not confirming the lock with a quorum of voters in this recipe.
• A fencing token. Nothing monotonic was given to PostgreSQL. The database will accept both writers.

Extend the TTL only with a renewal that checks the value, and only if you know the job can be longer than the TTL. Renewal has the same pause problem: if you fail to renew because you were paused, you must stop, and you will not know you must stop unless the next write is fenced.

Fencing tokens

Kleppmann’s argument is the one to remember in an interview and in a design review. The lock service increments a number every time it grants the lock. The holder includes that number on every write to the protected resource. The resource remembers the highest token it has accepted and rejects anything lower.

sqlCopy
update billing.invoice_export
set status = 'FINISHED', object_key = ?
where period = ? and fencing_token = ?;

If this returns zero rows, this worker is stale. It must not upload a second object, or it must upload only under a name the new token owns. Generating the token in Redis with INCR lock:invoice:2026-10:fence and writing it into this WHERE clause is a complete design. INCR alone, logged and then ignored, is not fencing.

The protected system has to cooperate. An S3 put does not understand fencing tokens. You fence in a database row that records the winning token, and you treat the object store as derived: write the object under a key that contains the token, then commit the row. A stale worker may upload an object nobody references. A lifecycle rule deletes unreferenced uploads. A stale worker must not overwrite the canonical key the new token already published.

Clock-based leases without a token cannot be repaired by “making the TTL longer.” A longer TTL widens the window where a crashed holder blocks everyone. A shorter TTL widens the window where a slow holder overlaps the next one. The token is what makes overlap safe.

Redlock, in one paragraph and a decision

Redlock acquires the single-instance recipe on a majority of independent Redis masters, with a wall-clock bound on how long acquisition took. It is an attempt to survive a single Redis node failing. It is not ZooKeeper and it is not Raft. Kleppmann’s 2016 analysis describes pause and clock scenarios in which two clients both believe they hold the lock. The Redis author’s rebuttal defends Redlock under different assumptions. You do not need to pick a winner in a blog argument to make an architecture decision: invoices, seats, and payments do not depend on Redlock. They depend on a constraint in the database or on a consensus system whose election you can name. A single Redis lease may still reduce duplicate work when the work is idempotent and a duplicate is cheap. That is an optimization. Label it as one.

ZooKeeper sessions and Curator

ZooKeeper’s primitive is the znode. An ephemeral znode exists until the client session ends. The session ends when the client closes it or when the ensemble has not heard from the client within the session timeout. A lock recipe creates an ephemeral sequential child under /locks/invoice-2026-10 and watches the predecessor. The client with the lowest sequence holds the lock. Apache Curator implements this as InterProcessMutex:

javaCopy
InterProcessMutex lock = new InterProcessMutex(client, "/tlp/locks/invoice-2026-10");
if (lock.acquire(5, TimeUnit.SECONDS)) {
  try {
    export(period);
  } finally {
    lock.release();
  }
}

acquire(long, TimeUnit) returns false on timeout. release() must run in a finally or the znode lives until the session dies. This is a better lease than Redis SET NX when you already operate ZooKeeper: session timeout is maintained by heartbeats, and the ensemble agrees on the znode. It is still a lease. A client paused longer than the session timeout loses the session; the ensemble deletes the ephemeral node; another client acquires; the paused client resumes and must not write. Curator will tell a careful client that the lock is lost if the session dies, via the connection state. A client that ignores ConnectionState.LOST and keeps exporting has recreated the Redis pause bug. The fence in PostgreSQL is still the backstop.

Do not set the session timeout below your worst GC pause. Do not set it to many minutes or a dead leader blocks the export. Measure pauses. This is the same TTL tradeoff, with a consensus system holding the lease.

Leader election is the lock applied to a role. Curator’s LeaderSelector calls takeLeadership on one client and holds leadership until that method returns. Use it when exactly one scheduler should enqueue jobs. Do not use it when any worker can take the next job; that is SKIP LOCKED. A leader that does the work itself becomes a bottleneck and a single point of failure for the work, even if the election is sound. Elect a scheduler. Let a pool of workers compete for rows.

What consensus is doing here

ZooKeeper’s Zab, etcd’s Raft, and a Postgres primary with synchronous replicas are different systems with a shared property: a majority agrees on a write before it is acknowledged, and a minority cannot grant a conflicting lock. etcd expresses a lease as a server-side TTL and a key attached to that lease. The key’s revision, or a compare-and-swap on that revision, is a natural fence if the database stores it. This lesson does not walk through a Java etcd client. The decision is whether you already run such a system. Standing up ZooKeeper only to lock an invoice job that Postgres can constrain is a new failure domain. Use the consensus system you already have for Kafka or for service discovery. Do not add one for a monthly export.

Kafka’s consumer group coordinator elects partition owners. That election is real and it is scoped to the group. Using a one-partition topic as a “cluster lock” couples your job to the log’s availability and rebalance behavior. Prefer the database. Lesson 33 already spent the group protocol on streaming.

Architecture and Design Diagrams

mermaidCopy
flowchart TD
  Need[Need exclusion] --> Row{Is the invariant a row?}
  Row -->|One export per period| U[Unique primary key]
  Row -->|Next job from a queue| S[FOR UPDATE SKIP LOCKED plus fencing_token]
  Row -->|No natural row| L{Do you already run ZK or etcd?}
  L -->|Yes| Z[Session lease and still fence the write]
  L -->|No| R[Redis lease only if duplicate work is harmless]
  U --> Fence[Writes check fencing_token or are idempotent]
  S --> Fence
  Z --> Fence

mermaidCopy
sequenceDiagram
  participant A as Worker A
  participant DB as PostgreSQL
  participant B as Worker B
  A->>DB: UPDATE fence = fence+1 WHERE lease expired RETURNING 7
  Note over A: GC pause longer than lease
  B->>DB: UPDATE fence = fence+1 RETURNING 8
  B->>DB: finish WHERE fencing_token = 8
  A->>DB: finish WHERE fencing_token = 7
  DB-->>A: 0 rows
  Note over A: A stops and discards its file

mermaidCopy
sequenceDiagram
  participant W as Worker
  participant RD as Redis
  participant DB as PostgreSQL
  W->>RD: SET key token NX PX 30000
  RD-->>W: OK
  W->>DB: write export
  Note over W,RD: TTL fires during write
  W->>RD: unlock script
  RD-->>W: 0 value mismatch
  Note over W,DB: Without a fence the export row may already be wrong

Trust boundary: lock paths and job tables are not public APIs. Redis ACLs and database grants allow the worker role to touch lock: keys or jobs.task and nothing else. A fencing token is not a secret; the random Redis lock value is. Do not log the Redis value in a way that lets another worker replay an unlock. Logging the fencing token is fine.

Technical Explanation

Crash matrix for the invoice job

The last row is why “we use Redis locks” is not an answer until you show the WHERE fencing_token = ? on the business write.

Reaper rules

A reaper that sets RUNNING back to READY without incrementing fencing_token reopens the row while the old worker still writes with the old token, and the WHERE clause cannot tell them apart. Increment first. Bound the reaper so it does not reclaim a job that is merely slow: locked_until is the worker’s renewed lease, set to now plus the next heartbeat interval, and the worker heartbeats by updating locked_until with the token in the WHERE clause. A heartbeat that does not check the token lets a stale worker keep a lease forever.

Do not run two reapers without SKIP LOCKED or a unique “reaper lock” row. Two reapers reclaiming the same job are the original bug one level up.

How long the work may hold a database transaction

SKIP LOCKED inside a transaction that also calls the PDF service holds a row lock and a connection for the whole call. Under load you exhaust the pool. Commit the transition to RUNNING first, do the slow work outside the transaction, then commit FINISHED with the fence. You traded “the row lock is the lock” for “the lease column is the lock.” That trade is correct when the external call is slow. It is incorrect if you then forget the fence.

Locks across services

A lock taken in the enrollment service to “protect” a write in the billing service does not include billing’s database. The holder can pause, the lock can expire, and billing still accepts the stale call. Put the unique constraint in the database that stores the invariant. If two services both think they own the invariant, you have a modeling problem, not a locking problem. Module 4’s service boundaries make this sharper: the owning service exposes an idempotent command. Callers retry. They do not hold a distributed lock across the HTTP call.

Clock and pause assumptions

Write them down, because the lock’s correctness is conditional on them:

• A process may pause for longer than any TTL you set. Full stop collections and debug sessions do this.
• Redis PX uses the Redis server’s clock for expiry. Client clocks do not expire the key. Client clocks do affect a lock recipe that computes “time remaining” locally, which Redlock does. The single-key PX recipe is safer on that one point and still unsafe on pause and failover.
• ZooKeeper session timeout uses the ensemble’s notion of time and heartbeats. A paused client does not heartbeat.
• NTP stepping a clock backward can extend or shorten leases in systems that compare timestamps in the client. Prefer server-side TTL and monotonic tokens over client-side “now + 30s” stored as a value you compare in Java.

Practical Implementation or Design Walkthrough

Problem

Generate one invoice export per billing period. The job may run for several minutes (PDF and object storage). Two schedulers may fire. Workers crash. A paused worker must not publish a second canonical file after a reaper has started a replacement.

Requirements

• At most one canonical object is referenced for 2026-10.
• A crash mid-export is retried.
• A late writer after a crash or a pause does not replace the canonical object.
• No new coordination cluster. PostgreSQL is already the billing store. Redis is already present for rate limits and is not required for this correctness.

Design

The unique primary key on period stops two inserts. The worker that inserts wins. fencing_token starts at 1. The worker writes the PDF to a temporary object key exports/2026-10/{token}.pdf, then:

sqlCopy
update billing.invoice_export
set status = 'FINISHED', object_key = ?
where period = ? and fencing_token = ? and status = 'RUNNING';

If the update count is 1, that object key is canonical. If it is 0, delete the temporary object or leave it for a lifecycle rule.

The reaper, once a minute:

sqlCopy
update billing.invoice_export
set fencing_token = fencing_token + 1,
    status = 'RUNNING',
    started_at = now()
where period = ?
  and status = 'RUNNING'
  and started_at < now() - interval '10 minutes'
returning fencing_token;

Ten minutes is above the p99 of a healthy export and below the time finance will wait. The old worker’s finish update matches the old token and writes zero rows. A second scheduler’s insert hits the primary key and does nothing.

SKIP LOCKED is the wrong tool for “one per period,” because there is nothing to queue. It is the right tool for the certificate jobs that are many independent rows. Do not force one pattern onto both.

Why Redis is optional here

A SET NX around the export would skip the second scheduler only when the lease is healthy. The unique key already does that for the whole lifetime of the row, not for 30 seconds. Adding Redis adds a failure mode (Redis down means either “skip the export” or “run unlocked”). When Redis is down, the database design still exports exactly once per winner. Choose the database.

Use Redis only if the critical section is a call to a system that cannot store a token and a duplicate call is expensive but not incorrect — and even then, prefer to make the call idempotent. A lock you cannot fence is a cost reducer. Say so in the ADR.

Tests

• Two threads insert the same period: one row, one unique violation.
• Finish with the wrong token: status stays RUNNING, update count 0.
• Reaper advances the token: the old worker’s finish does not set FINISHED; the new worker’s finish does.
• A test that pauses a thread is flaky. Simulate the pause by calling finish with the pre-reaper token after the reaper runs. That is the failure mode, without Thread.sleep.

Expected result

Finance downloads one object. invoice_export.object_key points at it. A retried worker either sees FINISHED and exits or loses the fence and discards its file. The runbook query is status = 'RUNNING' and started_at < now() - interval '10 minutes'.

Enterprise Real-World Scenarios

Scenario 1: Monthly invoice export

Business context. Finance expects one CSV and one PDF per calendar month, generated after midnight on the first. Two regions each run a scheduler because of an earlier active-active project. The export reads billing tables and writes an object.

Scale. One job per month per legal entity. Runtime two to eight minutes. Duplicate files cause a partner to import invoices twice.

Choices. Unique (legal_entity, period). Fencing token. Temporary object key containing the token. Reaper after ten minutes. Schedulers in both regions may both try; the primary key picks one. No Redis lock, no ZooKeeper. The second region’s database is a replica and is not allowed to start the export: only the writer primary runs the job, which you enforce by attempting the insert and treating a read-only transaction error as “not the primary,” not by a lock in each region.

What fails review. SET NX with a 30-second TTL and an eight-minute job. A shared “leader” flag in a cache with no fence. A reaper that resets status without incrementing the token.

Scenario 2: Certificate generation queue

Business context. Completing a course enqueues a certificate job. Thousands may be queued after a large cohort finishes. Any worker can run any job. Jobs call a PDF service that takes about two seconds.

Choices. jobs.task with SKIP LOCKED, status, locked_until, and fencing_token. Workers commit RUNNING before the PDF call and heartbeat locked_until with the token in the predicate. A reaper requeues expired leases and increments the token. The PDF is stored under an object key that includes the job id, overwritten by the winner; the finish update is fenced so a stale worker does not mark a reassigned job FINISHED after the new worker has started. Idempotent completion (lesson 37) means a second PDF for the same job id replaces the same object rather than creating certificate (1).pdf.

A Redis lock per job id would add a round trip and the pause bug to a queue Postgres already serializes. Curator would make sense if the organization already required a ZooKeeper lock for a resource that is not in Postgres. Certificates are in Postgres.

Module 4 split: the certificate service owns jobs.task. The course service enqueues by inserting a row or by sending a command the certificate service turns into a row. It does not take a lock “around” the certificate service. Enterprise Integration Patterns names the competing consumers. The exclusion is the row lock, not a shared mutex process.

Best Practices

• Ask whether a unique constraint can state the invariant. If it can, do not add a lock service.
• For a pool of workers, use FOR UPDATE SKIP LOCKED, commit the lease, do slow I/O outside the transaction, and fence every later write.
• Treat every TTL lock as a lease that can expire while the holder still runs. Carry a fencing token into the system of record.
• Release a Redis lock only with a compare-and-delete script. Use a random value. setIfAbsent with a Duration is the acquire.
• Do not adopt Redlock as the safety argument for money or inventory.
• If you use Curator, handle lost sessions and still fence the database write. Size the session timeout from measured pauses.
• Elect a scheduler if you must, and let workers compete for jobs. Do not do all the work inside takeLeadership.
• Do not hold advisory locks on pooled connections across requests.
• Keep the reaper’s reclaim predicate and the token increment in one statement.
• Document the crash matrix in the runbook: stuck RUNNING, late finish, double scheduler.

Common Mistakes and Anti-Patterns

Redis lock as the only correctness for a row you own. The database can reject the second insert. The Redis key can expire. You added a dependency and lost the constraint. Put the unique key on the period. Use Redis for rate limits, where approximation is the point.

DEL to unlock. After expiry, the lock belongs to someone else. A late DEL removes their lock. The Lua compare is the entire safety of the unlock. Skipping it because “the TTL is long” is how overlap becomes a guaranteed corruption window.

Fencing token generated and never checked. The worker logs token=8 and updates the row by primary key alone. The stale holder’s update still lands. The token has to be in the WHERE clause, and the application has to stop when the count is zero.

Reaper resets status and leaves the token unchanged. Old and new workers share a token. The fence cannot see the difference. Increment in the same update that requeues.

Holding FOR UPDATE during the PDF call. Connections run out, and the queue stalls while Postgres waits on a network call. Commit the lease, then call the PDF service.

Locking in service A to protect data in service B. The lock does not include B’s transaction. Make B’s operation idempotent and let B enforce its own constraint.

Redlock because the blog said “distributed.” Five Redis nodes and a wall-clock assumption are not a substitute for the unique key you skipped. If you need consensus, use the system that implements it and still fence external side effects.

Session timeout shorter than a GC pause, and no lost-session handler. ZooKeeper gives the lock to someone else. The paused JVM continues. Without a fence, both write. Measure pauses. Handle LOST. Fence the write anyway.

Troubleshooting and Design Review

Design review:

• What is the invariant, in one sentence, and which store owns it?
• Can a unique index or SKIP LOCKED express it?
• If a lease expires mid-work, which write becomes a no-op, and where is the token checked?
• What reclaims a crashed holder, and does it bump the token?
• What is the pause you measured, and how does it compare to the TTL or session timeout?
• What happens when the lock service is down? Does the system stop, or run without exclusion?
• Does any lock span two services or two databases?

Interview Questions and Answers

Q1. Why is a unique constraint often better than a distributed lock?

The constraint is enforced by the same transaction that writes the data. It does not expire because a garbage collector paused, and it does not disappear because a cache failed over. “One invoice export per period” is a primary key. The second worker gets a unique violation and stops. A lock in another system has to stay up, be released correctly, and still be checked against the row. Add a lock service when the resource cannot express the rule, not when it can.

Q2. What does SET key value NX PX 30000 actually promise?

If this Redis primary accepts the command, no other client can set that key until the value is deleted or the 30-second TTL elapses. The value must be an unguessable token so the unlock script can DEL only when GET still returns it. The promise ends at the TTL, at a failover that loses the key, and at any writer that does not honor the protocol. It does not include a fencing token. A holder that pauses past the TTL and then writes to a database that does not check a token will overlap the next holder.

Q3. What is a fencing token, and where must it be checked?

It is a number that increases every time the lease is granted. The holder sends it with each write. The system of record keeps the latest token it accepted and ignores a smaller one, typically UPDATE ... WHERE fencing_token = ?. Checking it only in the lock client is useless, because the stale client is the one that would have to notice, and it is stale precisely because it was paused or disconnected. The database, which did not pause, performs the check. Object stores that cannot compare tokens are written via a database row that records the winning key.

Q4. What goes wrong if you DEL a Redis lock when you are done?

Your TTL may have expired and another worker may have acquired the key. Your DEL removes their lock, and a third worker enters. The unlock has to be conditional on the value you set. That is the Lua script in the Redis lock documentation: if GET equals the token, DEL. If it does not, return 0 and treat yourself as no longer the holder. Then rely on the fence for any write you are about to make.

Q5. Why doesn’t this course use Redlock for invoices?

Redlock repeats the single-node lease on a majority of Redis masters and bounds acquisition with wall-clock time. It is a availability improvement for a cache-style lock, not a consensus protocol. Kleppmann’s critique describes pauses and clock behavior in which two clients both think they won. Whether every critique scenario is likely in your network is beside the point when a primary key already solves the invoice. Use Redlock only as an optimization around idempotent work, and do not cite it as the reason a double charge is impossible.

Q6. How does FOR UPDATE SKIP LOCKED assign jobs?

Workers select ready rows with FOR UPDATE SKIP LOCKED LIMIT n. Postgres locks those rows for the transaction and other workers skip them instead of blocking. The worker marks the row RUNNING, sets a lease and a new fencing token, and commits before any slow external call. A crash leaves RUNNING until a reaper expires the lease, increments the token, and makes the row ready again. The old worker’s later update matches zero rows because the token changed. Without the token increment, both workers are “the” owner.

Q7. What does a ZooKeeper ephemeral node add, and what does it not add?

The node exists for the life of the client session. If the client stops heartbeating for the session timeout, the ensemble deletes it and another client can acquire the lock. Curator’s InterProcessMutex is this recipe: acquire with a timeout, release in a finally. You get agreement among ZooKeeper nodes, which Redis SET NX on one primary does not give you. You do not get safety if the client ignores a lost session and keeps writing. Session timeout is still a lease. Fence the database write.

Q8. When would you elect a leader instead of locking each job?

When one process must decide, for example a scheduler that must not enqueue the same nightly plan twice, and the workers themselves can still compete. Curator LeaderSelector holds leadership for the duration of takeLeadership. If that method does every certificate, leadership is a bottleneck and a stall when that process pauses. Elect the scheduler. Assign the jobs with SKIP LOCKED. If the “leader” is a single row insert of the plan, you may not need an election at all: the unique key on the plan date is the election.

Q9. A service takes a lock, calls another service, and unlocks. What is wrong?

The lock does not cover the other service’s database. The call can succeed, the unlock can fail, and a retry takes the lock and calls again. Or the lock expires during the HTTP call and a second caller starts. Put idempotency and the unique constraint in the service that owns the write. The caller retries the command with the same key. A lock around a remote call feels like a transaction and does not have transactional semantics. Lesson 32’s saga is the long-running version of “there is no lock that spans these services.”

Q10. How do you test the pause-and-overlap failure without a flaky sleep?

Run the reaper’s update, which advances fencing_token, then call the first worker’s finish method with the old token. Assert the row did not become FINISHED and the new token’s finish does. Assert a second insert of the same period fails. Those are deterministic tests of the fence. A test that sleeps past a TTL in Redis is slow and still misses failover. You can test the unlock script with two tokens: the second SET NX fails while the first key exists, and a DEL script with the wrong token leaves the key in place. Use Testcontainers Redis for that script test. Do not mock SET to return success and call the design proven.

Hands-On Exercises

Exercise 1: Invoice export exclusion

Problem. Design the monthly export so two schedulers and a crash cannot publish two canonical files. Runtime is several minutes.

Tasks.

1. State the invariant and the table that enforces it.
2. Show the winning insert and the losing worker’s behavior.
3. Show the finish update with a fencing token.
4. Specify the reaper, including the token increment and the age threshold.
5. Explain where the PDF bytes go before the finish update commits.
6. Say what happens if Redis is unavailable, given your design.
7. List two crashes and the resulting row state.

Expected deliverables. Schema, three SQL statements, crash matrix, object-key rule.

Acceptance. A late finish with an old token updates zero rows. A second insert does not create a second period row. The design does not depend on a TTL shorter than the job.

Exercise 2: Review a Redis lock around seat assignment

The patch sets lock:seat:{id} with SET NX PX 5000, updates the seat row, and DELs the key. There is no token column. Write the review: pause, unlock, failover, and the unique constraint or fence you require instead. Seat assignment must not double-book.

Rubric.

Lesson Summary

A distributed lock with a timeout is a lease. Redis SET NX PX plus a compare-and-delete unlock is a lease on one primary. ZooKeeper ephemeral nodes are a lease tied to a session. Both can expire while a paused holder still believes it is the owner. A fencing token checked by the database makes that overlap a no-op. For TechLearningPro’s invoice export, a unique period key is stronger than either lease, and a reaper that increments the token retries a crashed winner safely. For a pile of certificate jobs, FOR UPDATE SKIP LOCKED assigns work and the same token protects the finish. Redlock is not the correctness argument for that data. Leaders are for singleton scheduling, not for doing the pool’s work inside the election. Across microservices, the owning service enforces the constraint and the caller retries an idempotent command. A lock wrapped around the HTTP call does not include the other database.

Further Reading

• Redis documentation, Distributed locks — SET with NX and PX, and the compare-and-delete unlock script.
• Martin Kleppmann, How to do distributed locking — fencing tokens and the Redlock critique.
• PostgreSQL documentation, SELECT ... FOR UPDATE (SKIP LOCKED) and advisory lock functions — pg_try_advisory_xact_lock.
• Apache Curator javadoc, InterProcessMutex (acquire(long, TimeUnit), release) and LeaderSelector.
• ZooKeeper programmer guide, ephemeral nodes and sessions.
• Next: Idempotency and Exactly-Once Processing, then Module 4 Microservices Architecture Principles and Enterprise Integration Patterns.