Message Queues

Introduction

Message queues decouple producers and consumers — producers publish messages; consumers process asynchronously at their pace. RabbitMQ, Amazon SQS, Azure Service Bus, and Redis lists enable workload buffering, spike absorption, and reliable background processing.

Unlike Kafka's log model, classic queues often delete messages after ack (point-to-point). HLD uses queues for email sending, image processing, order fulfillment steps, and retry with DLQ.

This lesson contrasts queues vs streams, delivery guarantees, and queue topology patterns.

Understanding the topic

Key concepts

• Point-to-point: one consumer per message (competing consumers).
• Pub/sub: broadcast copy to each subscriber queue/topic.
• At-least-once: ack after processing; duplicates possible — idempotent consumers.
• At-most-once: ack before process — loss possible, no duplicates.
• Dead-letter queue (DLQ): failed messages after max retries for inspection.
• Visibility timeout (SQS): message hidden while consumer works; extend if long job.

flowchart LR
  Producer --> Queue
  Queue --> Worker1
  Queue --> Worker2

Internal architecture

Architecture overview

flowchart LR
  Producer --> Queue
  Queue --> Worker1
  Queue --> Worker2

Step-by-step explanation

1. API accepts order → persists → publishes OrderCreated to queue → returns 202.
2. Worker pool consumes queue, processes payment capture, acks on success.
3. Retry with exponential backoff; poison message → DLQ after 5 attempts.
4. Priority queues for premium customers (RabbitMQ x-max-priority).
5. Scheduler cron publishes batch jobs to queue for horizontal workers.
6. Monitor queue depth — primary backpressure signal.

Informative example

Spring AMQP producer and consumer with manual ack and DLQ routing:

@Configuration
public class RabbitConfig {
    @Bean
    Queue orderQueue() { return new Queue("order.created", true); }

    @Bean
    DirectExchange orderExchange() { return new DirectExchange("shop"); }

    @Bean
    Binding binding() {
        return BindingBuilder.bind(orderQueue()).to(orderExchange()).with("order.created");
    }
}

@Service
public class OrderPublisher {
    private final RabbitTemplate rabbit;
    public OrderPublisher(RabbitTemplate rabbit) { this.rabbit = rabbit; }

    public void publish(OrderCreatedEvent event) {
        rabbit.convertAndSend("shop", "order.created", event);
    }
}

@Component
public class OrderWorker {
    @RabbitListener(queues = "order.created")
    public void handle(OrderCreatedEvent event, Channel channel,
                       @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException {
        try {
            process(event);
            channel.basicAck(tag, false);
        } catch (Exception e) {
            channel.basicNack(tag, false, false); // route to DLQ via policy
        }
    }
}

State queue depth alerts. Scale consumers horizontally when depth sustained high.

Real-world use

Real-world use cases

• E-commerce email receipts and invoice PDF generation.
• Healthcare HL7 message ingestion with retry and DLQ.
• Fintech nightly statement batch via SQS fan-out workers.
• Social video transcoding job queue.

Best practices

• Design idempotent consumers with business keys.
• Set visibility timeout > p99 processing time.
• Use DLQ and replay tooling — never silent drop.
• Limit message size; store payload reference in S3 for large blobs.
• Correlation ID in message headers for tracing.
• Back-pressure: slow consumers scale out, don't unbounded buffer forever.

Common mistakes

• No DLQ — poison message blocks queue forever.
• Processing before ack then crash — duplicate ok without idempotency.
• Single consumer for critical async path — no redundancy.
• Huge messages in queue — use claim check pattern.
• Using queue as database — messages expire, not queryable history.

Advanced interview questions

Summary

Message queues decouple and buffer asynchronous work. At-least-once delivery requires idempotent consumers. DLQ and visibility timeout are production essentials. Scale consumers with queue depth metrics. Kafka extends queues with durable log and replay — next lesson. Event streaming builds on queue concepts at higher throughput.