Idempotency

Introduction

An operation is idempotent if performing it multiple times has the same effect as once — critical for retries, webhooks, and distributed sagas. HTTP PUT and DELETE are idempotent by convention; POST is not unless you add an idempotency key.

Payment APIs (Stripe), order creation, and ticket booking all require idempotency in HLD. Store keys in Redis or SQL with response cache; return same result on duplicate within TTL window.

This lesson covers key generation, storage, expiration, and interview flows for safe mutations.

Understanding the topic

Key concepts

• Idempotency-Key header: client-generated UUID per logical operation.
• Server stores key → response mapping until TTL (24h typical payments).
• Processing states: IN_PROGRESS, COMPLETED — concurrent duplicates wait or return same.
• Natural idempotency: SET balance=100 vs INCR balance.
• Webhook delivery at-least-once requires idempotent handler by event ID.
• DB unique constraints as idempotency (order_id primary key prevents duplicate insert).

sequenceDiagram
  Client->>API: POST Idempotency-Key
  API->>Store: check key
  Store-->>API: not seen
  API->>API: process once
  API-->>Client: 201

Internal architecture

Architecture overview

sequenceDiagram
  Client->>API: POST Idempotency-Key
  API->>Store: check key
  Store-->>API: not seen
  API->>API: process once
  API-->>Client: 201

Step-by-step explanation

1. Client sends POST /payments with Idempotency-Key: uuid.
2. API checks idempotency_store; if COMPLETED return cached 201 body.
3. If new, insert IN_PROGRESS row, process payment, store response, mark COMPLETED.
4. Concurrent duplicate waits on lock or returns 409 IN_PROGRESS.
5. Redis SET key NX EX 86400 for fast path; PostgreSQL for audit durability.
6. Cleanup job archives keys older than retention policy.

Informative example

Idempotent payment endpoint with Redis lock and cached response:

@RestController
@RequestMapping("/api/v1/payments")
public class PaymentController {
    private final PaymentService payments;
    private final IdempotencyStore store;

    public PaymentController(PaymentService payments, IdempotencyStore store) {
        this.payments = payments;
        this.store = store;
    }

    @PostMapping
    public ResponseEntity<PaymentResponse> pay(
            @RequestHeader("Idempotency-Key") String key,
            @RequestBody PaymentRequest body) throws Exception {
        return store.execute(key, () -> {
            PaymentResponse result = payments.charge(body);
            return ResponseEntity.status(201).body(result);
        });
    }
}

@Component
public class IdempotencyStore {
    private final StringRedisTemplate redis;

    public <T> ResponseEntity<T> execute(String key, Callable<ResponseEntity<T>> action) throws Exception {
        String cacheKey = "idem:" + key;
        String cached = redis.opsForValue().get(cacheKey);
        if (cached != null) return deserialize(cached);
        Boolean acquired = redis.opsForValue().setIfAbsent(cacheKey + ":lock", "1", Duration.ofMinutes(5));
        if (Boolean.FALSE.equals(acquired)) throw new ConflictException("Request in progress");
        try {
            ResponseEntity<T> response = action.call();
            redis.opsForValue().set(cacheKey, serialize(response), Duration.ofHours(24));
            return response;
        } finally {
            redis.delete(cacheKey + ":lock");
        }
    }
}

Stripe-style pattern. For financial APIs combine Redis speed with PostgreSQL idempotency table for durability.

Real-world use

Real-world use cases

• Fintech card charges and wallet transfers.
• E-commerce order submit button double-click.
• Webhook handlers from payment provider and shipping carrier.
• Saga step retries in inventory reservation.

Best practices

• Require idempotency keys on all non-safe HTTP mutations from clients.
• TTL at least max retry window + client retry horizon.
• Return same HTTP status and body on replay.
• Use event ID dedup table for Kafka consumers.
• Unique DB constraints where natural (username, slug).
• Monitor duplicate key hit rate — indicates client bugs.

Common mistakes

• Idempotency only in memory — lost on restart, double charge.
• Different response body on replay — client confusion.
• Key scoped globally not per user — collision risk.
• No IN_PROGRESS handling — race double processing.
• Assuming GET retries always safe — GET with side effects exists.

Advanced interview questions

Summary

Idempotency makes retries and async delivery safe. Idempotency-Key header standard for payment and order APIs. Store key → response mapping with TTL and IN_PROGRESS state. Natural keys and unique constraints complement explicit keys. Webhook and Kafka handlers must deduplicate by event ID. Authentication establishes who calls idempotent APIs next.