Idempotency and Exactly-Once Processing

Advanced System Design & Software Architecture · Lesson 37 · High-Level Design and Distributed Systems

Lesson Overview

A learner double-clicks Enroll. The mobile client times out and retries. The acquirer approved the first call, and the response never reached TechLearningPro. Kafka redelivers PaymentAuthorized because the consumer committed its offset late. Three different hops, one business fear: two charges, two seats, two receipts. Exactly-once is the name people put on that fear. It is not a producer flag.

An operation is idempotent when performing it once and performing it several times with the same key leaves the same stored result. The tool is an idempotency key, scoped to the caller, stored with a hash of the request and the response you already returned. The transport may be at least once. The key makes the extra delivery a replay. Message Queues and Event Streaming established that at-least-once is the honest default. Distributed Locking and Coordination showed why a lease does not replace a unique key. This lesson puts the key on the payment HTTP API, in the transactional outbox, and in the consumer inbox.

Kafka’s exactly-once semantics are real and narrower than the phrase suggests. An idempotent producer plus a transaction (transactional.id, offsets committed with the output records, consumers on isolation.level=read_committed) gives exactly-once into Kafka for a consume-transform-produce flow. PostgreSQL, the acquirer, and the email provider are not inside that transaction. There is no Spring setting that enlists JdbcTemplate and KafkaTemplate in one atomic commit. The end-to-end design is: one database transaction writes the business row and the outbox; a relay publishes at least once; each consumer inserts an inbox row in the same transaction as its own write. That composition is what you can defend in a design review. Calling it “Kafka exactly-once” without the inbox is how a second seat gets created.

The HTTP shape follows the same idea. POST /payments requires an Idempotency-Key. The same principal, the same key, and the same body return the stored status and body, including a stored decline. The same key and a different body is 409 Conflict. Clients retry 409-in-progress, 429, and 503 with that same key, which is the contract Rate Limiting, Throttling and Load Shedding already asked for. Module 4’s services keep this contract at every command boundary. API Design, Versioning and Contracts publishes the header. Microservices Architecture Principles is why you cannot share one database transaction across those services and must dedupe instead.

Learning Objectives

By the end of this lesson, you should be able to:

1. Define idempotency as “same key, same stored result,” including declines and error responses you intend to replay.
2. Design an Idempotency-Key scoped to the principal, with a request hash, IN_PROGRESS and COMPLETED states, and 409 on a hash mismatch.
3. Place the acquirer call outside a long database transaction and still retry it safely when the acquirer documents an idempotency mechanism.
4. Separate Kafka’s transactional exactly-once (log in, log out) from exactly-once side effects in PostgreSQL or at an HTTP provider.
5. Write the outbox row in the same transaction as the payment update, and the inbox row in the same transaction as the consumer’s update.
6. Say what enable.idempotence and transactional.id each cover, and what a new process that sends again does not cover.
7. Reconcile with the acquirer when a result is UNKNOWN, instead of inventing a success.
8. Carry the pattern to microservices: every command that can be retried across a process boundary has a key the owner stores.

Prerequisites

• Lesson 22’s payment case study: the gateway port, the rule that a timeout is not an approval, and a unique (principal, idempotency key). This lesson is the distributed version of that record, including the outbox and the consumer.
• Lesson 9: the outbox exists because a commit and a KafkaTemplate.send are two systems. Lesson 33: consumers see duplicates. You need both chapters’ conclusions.
• Lesson 32, Distributed Transactions and the Saga Pattern: a saga step is a retried command. The step key is an idempotency key.
• Lesson 36: a lock does not make a payment exactly-once. The unique key does.
• RFC 9110’s definition of idempotent methods. PUT and DELETE are idempotent when implemented as the standard says. POST /payments is not, unless you add a key.

Detailed Core Concepts

What “once” modifies

Say the effect you mean.

“Exactly-once processing” in this course means the end-to-end row: the learner is charged once, the seat exists once, and a replay returns the original result. It is a property of the keys and the transactions you control. It is not a checkbox on the broker.

The HTTP idempotency record

Stripe popularized Idempotency-Key on charges. The IETF draft draft-ietf-httpapi-idempotency-key-header specifies a header of that name. Treat the header as a contract you publish, not as a library. Our rules:

• The header is required on POST /payments. Missing header: 400, no charge.
• The key is opaque, at most 255 characters, generated by the client. A UUID is a good client choice. The server does not derive it from the amount, or two different purchases that happen to match get collapsed.
• Scope is (principal_id, idempotency_key). Two tenants may use the same UUID. One tenant must not be able to replay another tenant’s key. Unauthenticated callers do not get this endpoint.
• Store a hash of the canonical request: method, path, and a stable JSON encoding of the fields that define the operation (amount, currency, offering id, payment-method token reference). Whitespace and key order must not change the hash. Exclude the header itself from the body hash.
• Same principal, same key, same hash, state COMPLETED: return the stored status and body. Do not call the acquirer again.
• Same principal, same key, different hash: 409. The client is reusing a key for a new operation. They must send a new key.
• Same principal, same key, same hash, state IN_PROGRESS: 409 with Retry-After: 1 if the attempt is still fresh, so a double-click does not start a second charge. If the attempt is older than the acquirer timeout and still IN_PROGRESS, the retry resumes that row. It does not insert another payment.

Store the decline. A replay of a declined charge returns the decline. If you only stored successes, a retry after a decline would call the acquirer again and might succeed on a second card attempt the client did not intend, or it might decline again and look like a new decision. The stored body is the decision you already made.

Keep the row at least as long as clients retry. Seventy-two hours covers mobile retries and support-driven replays. After expiry, the same key is a new request. Document the TTL in the API. A client that reuses keys after a week will create a second charge and it will be obeying the contract you wrote. Choose a TTL finance agrees with.

Where the external call sits

A single database transaction cannot hold the acquirer.

textCopy
begin
  insert idempotency IN_PROGRESS
  insert payment PENDING
commit
call acquirer with the same key
begin
  update payment to AUTHORIZED or DECLINED
  update idempotency COMPLETED with status and body
  insert outbox PaymentAuthorized or PaymentDeclined
commit

If you crash before the first commit, there is no charge and no row. The client retries and starts clean. If you crash after the acquirer returns and before the second commit, the row is still IN_PROGRESS. The resume path calls the acquirer again with the same key. That is safe only if the acquirer’s documented API deduplicates that key and returns the original approval or decline. Use the header or field their documentation names. Do not invent a vendor endpoint. If the acquirer has no idempotency mechanism, you cannot resume blindly: set UNKNOWN, do not capture a second time, and reconcile with their settlement or status query if they document one. UNKNOWN is a state. Guessing AUTHORIZED is how you deliver a course the learner was not charged for, and guessing DECLINED is how you charge them and show a failure.

The first transaction is short. It does not contain the HTTP call. Lesson 22’s rule stands. The idempotency row is what makes the gap between the two transactions recoverable.

Concurrent double-clicks both try the first insert. One wins. The other reads IN_PROGRESS and returns 409 with Retry-After. It does not wait inside the servlet for the winner, and it does not start a second PENDING payment. The client’s retry then hits COMPLETED and renders the stored body.

Outbox and inbox

The outbox row is inserted in the second transaction, next to the payment update. A crash there rolls the payment update back too, so you do not publish an authorization you did not store. The relay sends later, at least once, and sets published_at only after KafkaTemplate.send(...).get(...) succeeds. A crash after the broker ack and before published_at sends a duplicate record. That is expected.

The consumer’s inbox:

sqlCopy
create table enrollment.inbox (
  consumer text not null,
  event_id uuid not null,
  primary key (consumer, event_id)
);

Insert the inbox row and the seat in one transaction. A unique violation means this event_id was already applied. Commit nothing else, return, and let the offset advance. Deduping on the Kafka offset instead of event_id breaks when a new group replays, or when the outbox relay produces a second offset for the same event. The event id was chosen when the outbox row was inserted. It is stable across redelivery.

The inbox and the business write must share a transaction. An inbox insert that commits, then a seat insert that fails, loses the seat and suppresses the retry. A seat insert that commits, then an inbox insert that fails, creates a duplicate on retry. Same transaction, same commit.

Kafka transactions, scoped

Use a Kafka transaction when the consumer reads from one topic and writes to another, and both the output and the input offset must become visible together. Spring Kafka: DefaultKafkaProducerFactory.setTransactionIdPrefix("enrollment-tx-"), a KafkaTransactionManager, and the listener container’s setKafkaAwareTransactionManager. The prefix is how Spring assigns a transactional.id per producer. Do not also pin a single transactional.id in the property map or every instance will fence the others on startup, which is the actual fencing behavior of Kafka transactions: a new producer with the same transactional id bumps the epoch and fences the old one. That fencing is about producers, not about your payment row.

Consumers that must not see aborted transaction markers set isolation.level=read_committed (ConsumerConfig.ISOLATION_LEVEL_CONFIG). The default is read_uncommitted, which can expose records from a transaction that later aborts. For a pipeline that is entirely inside Kafka, these settings are the exactly-once feature the documentation describes.

They do not wrap the inbox insert. If you turn them on and write to PostgreSQL inside the listener, you have two commits again. A failure between them is the dual-write lesson 9 forbade. Keep the payment path on the outbox. Use Kafka transactions for stream-to-stream work, such as projecting an internal topic to a partner topic, where the output is still Kafka.

enable.idempotence=true (the producer default since Apache Kafka 3.0, and implied by transactions) deduplicates producer retries within the session. It is necessary for sane transactions. It is not the inbox.

Reconciliation

Idempotency reduces duplicates. It does not prove the acquirer and your database match. A daily job compares the acquirer’s settlement file to payments.payment. Rows you have as AUTHORIZED that they do not show, and captures they show that you have as UNKNOWN or missing, become a queue for a human or a documented repair command. The repair command itself takes an idempotency key. Reconciliation is the backstop for bugs, for acquirers without idempotent retry, and for the UNKNOWN state. Skipping it because “we enabled exactly-once” is how a mismatch sits unnoticed until finance month-end.

Architecture and Design Diagrams

mermaidCopy
sequenceDiagram
  participant C as Client
  participant P as Payment service
  participant DB as PostgreSQL
  participant A as Acquirer
  participant K as Kafka
  participant E as Enrollment
  C->>P: POST /payments Idempotency-Key
  P->>DB: insert IN_PROGRESS and PENDING
  P->>A: charge with the same key
  A-->>P: approved
  P->>DB: COMPLETED, payment AUTHORIZED, outbox row
  Note over P,DB: one commit
  P-->>C: 201 stored body
  C->>P: retry same key
  P->>DB: read COMPLETED
  P-->>C: stored 201, no acquirer call
  DB->>K: relay send
  K->>E: deliver
  E->>DB: inbox event_id and seat, one commit

mermaidCopy
flowchart TD
  R[Retry or redelivery] --> H{Same key and same hash?}
  H -->|No key match| New[New operation]
  H -->|Key match, hash differs| C409[409 Conflict]
  H -->|COMPLETED| Replay[Return stored response]
  H -->|IN_PROGRESS fresh| Busy[409 Retry-After]
  H -->|IN_PROGRESS stale| Resume[Resume acquirer with same key]
  Resume --> U{Acquirer supports the key?}
  U -->|Yes| Store[Store their original result]
  U -->|No| Unk[Mark UNKNOWN and reconcile]

mermaidCopy
flowchart LR
  subgraph inKafka [Exactly-once inside Kafka]
    IN[Input topic] --> TX[Kafka transaction]
    TX --> OUT[Output topic]
    TX --> OFF[Offset commit]
  end
  subgraph endToEnd [End-to-end side effects]
    OB[(outbox in Postgres)] --> RELAY[Relay at least once]
    RELAY --> TOPIC[Topic]
    TOPIC --> INBOX[(inbox plus business row)]
  end

Trust boundary: the idempotency key is not a secret, but the stored body can contain an acquirer reference and the last four digits if you chose to keep them. It must not contain a PAN or a CVV; lesson 22 kept those out of the process entirely. The payment-method token in the request hash is a reference, not a card number. Logs contain the principal id and the idempotency key, not the token and not the bearer credential. Enrollment’s inbox stores the event id and the seat. It does not store the payment-method token even if a careless producer put one on the event. Reject that payload to the dead letter.

Technical Explanation

Canonical hash

Serialize the operation fields with sorted keys and no insignificant whitespace. Compute SHA-256. Store the raw bytes. Comparing JSON strings without a canonical form treats {"a":1,"b":2} and {"b":2,"a":1} as different operations and returns 409 on a harmless retry from a client library that reorders keys. That 409 is a client bug you caused. Canonicalize.

Do not include volatile fields: client timestamp, a random request id that changes per attempt. If those are in the hash, every retry looks like a different operation. The idempotency key is the identity. The hash is the payload check.

Status codes worth fixing in the client guide

402 is not a requirement. Pick a documented decline status and store it. The rule is “replay the bytes,” not “map old declines to 200.”

A gateway timeout (lesson 34) can happen after you stored COMPLETED. The client retries and gets the stored body. That is why the key exists. A gateway timeout before the first commit looks like a failed request; the retry starts or resumes. Both are safe if the client keeps the key.

What not to dedupe on

• Kafka offset. It changes when the relay sends twice and when a group is recreated.
• In-memory set. It vanishes on deploy, which is when redelivery happens.
• Redis SET NX with a short TTL. Lesson 36: the key expires, and a late duplicate applies. A TTL cache in front of the inbox is an optimization. The inbox primary key is the guarantee. If Redis loses the key you must still hit PostgreSQL.
• A lock around the handler. The lock can expire. The unique key cannot, until you delete the row on purpose at the retention horizon.

Transactional ids fence producers, not business duplicates

Starting a second process with the same transactional.id fences the first producer. That prevents two relays from using one transactional identity, which would corrupt Kafka’s transaction state. It does not notice that two different transactional ids both published the same event_id. The consumer’s primary key notices. Give each relay instance its own id via Spring’s prefix (the factory appends a suffix). Do not copy one transactional.id into every pod’s environment.

Sagas and the step key

Lesson 32’s refund calls enrollment, then payments, then email. Each call carries Idempotency-Key derived from refundId and the step name, not a fresh UUID per HTTP attempt. A fresh UUID per attempt is a second refund. The saga store records the step as done only after the stored response says so. A timeout leaves the step in progress and the retry reuses the key. Compensations have their own keys. “Undo” is not a second execution of “capture.”

Multi-service reality

Payment and enrollment will not share a transaction once they are separate services. The outbox is how payment publishes. The inbox is how enrollment consumes. A synchronous HTTP command from payment to enrollment is the other legal shape, and it needs the same key on the request. Do not do both “HTTP call inside the payment transaction” and “outbox of the same fact” or you will enroll twice by two paths. Pick one. This lesson’s payment path publishes; enrollment does not also get a direct call.

Practical Implementation or Design Walkthrough

Problem

POST /payments charges a learner for an offering. Clients retry on timeout. The acquirer’s API documents an idempotency header (use the name in their docs). Enrollment must create one seat when the payment is authorized, via Kafka, and must tolerate duplicate records.

Schema

sqlCopy
create table payments.idempotency (
  principal_id uuid not null,
  idempotency_key text not null,
  request_hash bytea not null,
  state text not null,
  payment_id uuid,
  response_status int,
  response_body jsonb,
  created_at timestamptz not null default now(),
  primary key (principal_id, idempotency_key)
);

create table payments.payment (
  payment_id uuid primary key,
  principal_id uuid not null,
  offering_id uuid not null,
  amount_cents int not null,
  state text not null,
  acquirer_ref text
);

create table payments.outbox (
  event_id uuid primary key,
  aggregate_id uuid not null,
  event_type text not null,
  payload jsonb not null,
  created_at timestamptz not null default now(),
  published_at timestamptz
);

state on the idempotency row is IN_PROGRESS or COMPLETED. Payment state includes PENDING, AUTHORIZED, DECLINED, UNKNOWN.

Handler outline

javaCopy
public Response replayOrStart(Principal principal, String key, byte[] hash, PaymentRequest body) {
  InsertResult inserted = store.tryInsert(principal.id(), key, hash);
  if (inserted.conflict()) {
    IdempotencyRow row = store.load(principal.id(), key);
    if (!row.hashEquals(hash)) {
      return Response.conflict("idempotency key reused with a different request");
    }
    if (row.completed()) {
      return row.storedResponse();
    }
    if (row.fresh()) {
      return Response.inProgress();
    }
    return resume(row, body);
  }
  PaymentId id = store.insertPending(principal.id(), body);
  AcquirerResult result = acquirer.charge(key, body);
  if (result.unknown()) {
    store.markUnknown(id);
    return Response.unknown(key);
  }
  return store.complete(principal.id(), key, id, result);
}

tryInsert and insertPending commit before acquirer.charge. complete updates the payment, marks the idempotency row COMPLETED with the HTTP body, and inserts the outbox row, in one transaction. resume calls acquirer.charge with the same key and then complete. fresh means created_at is within the acquirer timeout window, so a concurrent duplicate waits via 409 rather than starting a second in-flight call. A crashed process leaves a row that stops being fresh, and the retry resumes.

store.complete is the only writer of AUTHORIZED. It is also the only writer of the outbox event for that payment. A unique constraint on outbox.aggregate_id plus event_type for this flow stops a bug from inserting two authorization events if complete is attempted twice. The idempotency state transition IN_PROGRESS -> COMPLETED with WHERE state = 'IN_PROGRESS' makes the second complete a no-op.

Consumer

javaCopy
@KafkaListener(topics = "payments.payment-authorized", groupId = "enrollment")
public void onAuthorized(ConsumerRecord<String, String> record) {
  PaymentAuthorized event = json.read(record.value());
  enrollment.apply(event);
}

void apply(PaymentAuthorized event) {
  transaction.executeWithoutResult(status -> {
    inbox.insert("enrollment", event.eventId());
    seats.insertIfAbsent(event.offeringId(), event.principalId(), event.paymentId());
  });
}

inbox.insert throws a duplicate-key exception you translate into “already applied,” and you must roll back the seat insert with it — which the shared transaction does if you rethrow after a failed insert, or if you insert inbox first and stop. Catch the unique violation inside the transaction, do not insert the seat, and complete the transaction empty so the listener returns normally and the offset commits. An empty commit is correct: the work is already durable.

Seat uniqueness on (offering_id, principal_id) is a second fence for the business invariant “one seat.” The inbox fences the event. You want both. Two different event ids must not create two seats; the seat key stops that. Two deliveries of one event id must not create two seats; the inbox stops that even if the seat insert were not unique.

Tests

• Same key, same body, twice: one row in payments.payment, one acquirer call, identical response bytes.
• Same key, different amount: 409, one payment row.
• Crash stub: first transaction committed, acquirer double returns the same approval, complete runs once: one payment AUTHORIZED, one outbox row.
• Acquirer returns an unknown timeout and documents no idempotent replay: payment UNKNOWN, no outbox authorization event.
• Consumer apply twice with one eventId: one seat.
• Two event ids for one principal and offering: one seat, and a visible error or a no-op on the second, according to the unique constraint. The test chooses “no second seat.”

Expected result

The client can retry until it receives the stored 201. Enrollment sees one seat. A duplicate Kafka record does not add another. Finance can join payment to the acquirer file. UNKNOWN rows are a report, not a silent success.

Enterprise Real-World Scenarios

Scenario 1: Mobile enrollment at the edge of the network

Business context. Learners enroll from phones on unreliable networks. The app times out at 10 seconds and retries three times. The acquirer often responds in under two seconds, and sometimes the response dies at the gateway. Double charges are a support emergency. Double seats fill a cohort incorrectly.

Scale. Launch day, a few hundred enrollment POSTs per second at peak, far below the read traffic in lesson 35. The idempotency table is small. The correctness matters more than the QPS.

Choices. Client generates a UUID when the learner taps Enroll and reuses it for every retry of that tap. A new tap is a new key. The payment service stores hash, state, and body for 72 hours. The acquirer’s documented idempotency header receives the same key. The outbox publishes PaymentAuthorized with a stable eventId. Enrollment’s inbox and the seat unique key apply it once. The app treats stored 201, 409 with Retry-After, 429, and 503 as “same key.” It treats 409 with the hash-mismatch body as “stop, this is a bug,” not as “new key, try again,” because a new key would charge twice for a request the user thinks is one purchase. Only a new user intent mints a new key.

What fails review. A new UUID per HTTP attempt. A Redis lock around the charge with a 5-second TTL. Producer idempotence enabled and the inbox table deleted as redundant. UNKNOWN mapped to success so the UI looks calm.

Scenario 2: Partner batch that replays a file

Business context. A university uploads a CSV of enrollments through the partner API from lesson 34. The job fails halfway and the partner resubmits the file the next morning. Some rows were charged. Some were not.

Choices. Each row has an idempotency key the partner owns, stable across resubmits: the university’s student id, the offering id, and the term, or a UUID they persist in their system. Your API does not mint a new key per upload. Replays of completed rows return the stored body and do not call the acquirer. Rows that never arrived are new. Rows left IN_PROGRESS past the timeout resume. The partner contract in lesson 40 documents the header, the 409 cases, and the 72-hour retention. A partner who generates a new key per file row on each upload will double-charge, and the documentation has to say that in the first page, not the appendix.

The microservices bridge is the ownership of the key. Payment owns the charge record. Enrollment owns the seat and the inbox. Neither database is shared. CQRS and Event Sourcing may later project these events into a read model; the projection is another consumer with its own inbox, not a second charger. Enterprise Integration Patterns calls the inbox a deduplicating consumer. The name is optional. The unique key is not.

Best Practices

• Require an idempotency key on every retried POST that creates money, a seat, or a message. Scope it to the principal.
• Hash a canonical form of the request. Mismatch is 409. Match and COMPLETED is a stored replay, including declines.
• Commit IN_PROGRESS before the external call. Complete, store the body, and insert the outbox in one later transaction.
• Resume a stale IN_PROGRESS only when the acquirer deduplicates the key. Otherwise mark UNKNOWN and reconcile.
• Deduplicate consumers on a stable event id in the same transaction as the business write. Keep a unique constraint on the business invariant too.
• Use Kafka transactions for Kafka-to-Kafka pipelines. Do not describe them as covering the database.
• Derive saga step keys from the saga id and the step name. Never mint a new key because the HTTP client retried.
• Retain records for the full client retry window, and publish the TTL.
• Log keys and principal ids. Do not log payment-method tokens or PANs.
• Reconcile with the acquirer on a schedule. Idempotency is not a substitute for that file.

Common Mistakes and Anti-Patterns

New idempotency key on every retry. The header is present, so the code looks compliant. Each timeout creates a new payment. The key must be stable for the user intent. Generate it at the tap or the file row, and persist it on the client before the first attempt.

Storing only successes. A declined charge is retried as if it never happened, or a timeout after a decline opens a second attempt. Store the response you returned, whatever the status was, once the outcome is known.

Inbox committed separately from the seat. Either the retry is suppressed and the seat is missing, or the seat exists and the retry adds another. One transaction. The unique violation path rolls back or inserts nothing.

Deduping on offset or on a TTL cache alone. Redeployment and replay create new offsets and empty caches. The primary key on (consumer, event_id) is durable for the life of the event. A cache may sit in front of it.

“We set enable.idempotence=true, so delete the inbox.” That flag deduplicates producer retries inside one session. The outbox relay can still append the same payload twice. The consumer must dedupe. Kafka transactions tighten the Kafka-to-Kafka case. They still do not commit PostgreSQL.

Calling the acquirer inside the database transaction. Connections stay checked out for the HTTP round trip, and a rollback after the acquirer approved leaves a charge with no row. Commit the intent, call, then commit the result.

Two paths for the same fact. The payment service commits, calls enrollment over HTTP, and also publishes PaymentAuthorized. Enrollment applies both. One path, one key. The other path is how “exactly-once” becomes “twice, reliably.”

Mapping UNKNOWN to approved so the UI is simple. The learner enters the course without a capture, or you capture later and surprise them. UNKNOWN is visible, retryable under the rules above, and on the reconciliation report.

Troubleshooting and Design Review

Design review for a new command:

• What is the idempotency key, who generates it, and how long is it stored?
• What is the hash input, and what is deliberately excluded?
• What is returned for replay, mismatch, and in-progress?
• Where is the external call relative to commit?
• If the external system has no idempotency, what is the UNKNOWN path?
• If a fact is published, where is the outbox insert relative to the state change?
• What does the consumer use as a durable dedupe key, in which transaction?
• What reconciliation proves the edges still match?

Interview Questions and Answers

Q1. What does idempotent mean for POST /payments?

The method is not idempotent in the HTTP specification. You make the operation idempotent with a key. The same principal, the same Idempotency-Key, and the same canonical body always produce the same stored payment and the same response bytes, whether the client called once or ten times. That includes a stored decline. A different body with the same key is a conflict, not a second charge. A new key is a new operation. The key is scoped to the principal so tenants do not collide.

Q2. Why is exactly-once an end-to-end claim rather than a broker setting?

The learner’s “once” means one capture and one seat. Kafka can prevent duplicate records inside a consume-transform-produce transaction when the producer has a transactional.id, offsets are part of that transaction, and consumers use read_committed. The acquirer and PostgreSQL are not in that transaction. At-least-once delivery plus a unique idempotency key is the claim you can implement across those systems. Saying “exactly-once” without naming the key and the store is a slogan. Name the key, the transaction boundaries, and the replay response.

Q3. What do you persist on the idempotency row, and why the hash?

The principal, the key, a hash of the canonical request, a state, and, when completed, the status and the body. The hash detects a client that reused a key for a different amount or offering. Without it, the second request would return the first payment and the client would believe the new amount was charged. With it, they receive 409 and must choose a new key for a real new purchase. Store the body so the replay does not re-execute. Store declines so a retry does not look like a fresh attempt.

Q4. How do you handle two concurrent requests with the same key?

The first insert wins because of the primary key. The second observes IN_PROGRESS. If the row is younger than the acquirer timeout, return 409 and Retry-After and do not call the acquirer. The client’s retry then receives the stored result if the first request completed. If you let both call the acquirer, you depend entirely on the acquirer to collapse them, and you may store two payment ids. One row, one payment id, created in the first transaction.

Q5. Where does the acquirer call sit relative to the database transaction?

Not inside it. Commit IN_PROGRESS and PENDING, call the acquirer, then commit COMPLETED, the payment state, and the outbox row together. A crash in the gap resumes by calling the acquirer with the same key, if their API deduplicates it. If it does not, mark UNKNOWN and reconcile. Holding the database transaction open during the HTTP call exhausts the pool and can roll back a local row after the acquirer has already approved.

Q6. What is the difference between the outbox and the inbox?

The outbox is the producer’s durable intent to publish, written in the same transaction as the state change, so you do not lose the fact and you do not publish a fact you rolled back. The relay is at least once. The inbox is the consumer’s record that this event id was applied, written in the same transaction as the consumer’s own write, so a redelivery does not apply twice. You need both. The outbox does not dedupe the consumer. The inbox does not save a fact that was never written down.

Q7. What does enable.idempotence fail to do?

It stops the producer from appending the same batch twice when it retries within its session, given acks=all and the in-flight limit the client enforces. It does not remember that your outbox relay successfully sent an event and then crashed before recording published_at. That second send is a new append. It does not dedupe two business events. Leave the consumer’s unique key in place. Turn the flag on because producer retries without it create yet another class of duplicate, not because it replaces the inbox.

Q8. When are Kafka transactions the right tool?

When input and output are both Kafka, and you need the output records and the input offsets to commit or abort together. Configure a transactional producer (setTransactionIdPrefix on DefaultKafkaProducerFactory), run the listener with KafkaTransactionManager, and set consumer isolation.level to read_committed. A distinct transactional.id per instance avoids fencing your own live pods; the same id on two pods is a bug. Do not use this configuration as a substitute for an outbox when the source of truth is PostgreSQL. The database commit and the Kafka commit are still two commits.

Q9. A saga step times out. What key does the retry use?

The same key as the first attempt, derived from the saga id and the step name. The step stays in progress until a stored response says it completed. A new UUID on the retry is a second capture or a second seat. Compensating actions get their own keys so a replay of the compensation does not become a second refund, and so a replay of the forward step is not confused with the undo. Lesson 32 owns the state machine. This lesson owns the key on each command.

Q10. How do you prove the design in tests without a real acquirer?

A fake acquirer that approves a key once and returns the same approval on the second call with that key. Test one: two HTTP calls, one fake invocation, identical bodies. Test two: hash mismatch, zero additional fake invocations, 409. Test three: the store commits PENDING, the fake approves, and you drop the process before complete; a resume calls the fake again and complete leaves one AUTHORIZED row and one outbox row. Test four: apply of the same eventId twice inserts one seat. Test five: the fake returns unknown and you assert UNKNOWN and no authorization event. A mock that never checks the key will pass a handler that generates a new acquirer request per retry.

Hands-On Exercises

Exercise 1: Idempotent payment and one seat

Problem. Specify POST /payments and the enrollment consumer so retries and Kafka redelivery create one charge and one seat.

Tasks.

1. Define the idempotency table, the hash inputs, and the TTL.
2. Specify status codes for replay, mismatch, in-progress, and unknown.
3. Place the acquirer call relative to the two commits. State what you require from the acquirer’s documentation.
4. Show the outbox insert and the relay’s published_at rule.
5. Show the inbox and seat constraints and the transaction around them.
6. Say what enable.idempotence and a Kafka transaction do not replace.
7. Define the client rule for when a new key may be minted.

Expected deliverables. Schema, sequence, status table, consumer transaction note, client rule.

Acceptance. A retried POST with the same key does not call the acquirer after COMPLETED. A redelivered event does not create a second seat. A hash mismatch does not return another learner’s payment. UNKNOWN does not publish PaymentAuthorized.

Exercise 2: Review “exactly-once Kafka” on the charge path

The change sets enable.idempotence=true, publishes with KafkaTemplate after committing the payment, and removes the inbox because “the broker is exactly-once.” The client generates a new idempotency key inside the HTTP interceptor on each attempt. Write the review. Name the duplicates that remain and the minimum design you would put back.

Rubric.

Lesson Summary

Exactly-once, for a charge and a seat, is an idempotency key plus transactions you actually control. The client keeps one Idempotency-Key for one intent. The server stores the hash and the response, scoped to the principal, and replays it, including declines. The acquirer is called between two short commits, with their documented idempotency field, or the payment stays UNKNOWN and reconciliation picks it up. The outbox is committed with the authorization. The relay is at least once. Enrollment’s inbox and a unique seat are committed together, so a redelivery is a no-op. Kafka’s idempotent producer and transactions make Kafka-to-Kafka pipelines exactly-once inside the log. They do not commit PostgreSQL, and they do not charge a card. Producer idempotence stays on. The inbox stays too. When payment and enrollment are separate services, this key and these two tables are the contract that replaces a shared transaction. That is the handoff to the microservices module: commands are retried, owners dedupe, and nobody reaches for a distributed lock to fake atomicity.

Further Reading

• RFC 9110, idempotent methods, for what PUT and DELETE already mean before you invent a key for POST.
• IETF draft The Idempotency-Key HTTP Header Field (draft-ietf-httpapi-idempotency-key-header). A draft: read the current revision’s requirements before you depend on a detail.
• Stripe’s API reference on idempotent requests, as a widely deployed description of “same key, same parameters, stored result.” Follow their header only as a public example of the idea; your acquirer’s field names come from your acquirer’s docs.
• Apache Kafka documentation on idempotent producers and transactions (transactional.id, isolation.level).
• Spring for Apache Kafka, transactions — setTransactionIdPrefix, KafkaTransactionManager.
• Microsoft, Transactional outbox and idempotency patterns.
• Next in Module 4: Microservices Architecture Principles, API Design, Versioning and Contracts, CQRS and Event Sourcing, Enterprise Integration Patterns.