Message Queues: Async Processing at Scale

With multiple servers processing jobs in parallel, a server can crash mid-job. The jobs it was holding are now unfinished — they must not be lost, and they must be reassigned.

The Reassignment Pattern

A monitor/notifier watches server health. On crash detection:

1. Fetch the crashed server’s unfinished jobs from shared storage.
2. Redistribute them to alive servers.
3. Distribution uses consistent hashing via the load balancer.

Because consistent hashing maps the same job ID to the same server, jobs already running on healthy servers don’t get duplicated — only the dead server’s slice moves.

This pattern — durable job list, health checks, retry, redistribution — is essentially how a message queue works.

What a Message Queue Gives You

• Producers enqueue work; consumers process it. Neither knows about the other directly.
• Messages are persisted until acknowledged, so crashes don’t lose work.
• Consumers scale independently of producers (add workers at peak, drain backlog off-hours).
• Failed messages can be retried or routed to a dead-letter queue.

The Pub-Sub Model

Pub-sub generalizes the queue from one consumer per message to many:

• Publishers emit events (“order placed”).
• A broker (Kafka, RabbitMQ) routes them to topic subscribers.
• Subscribers consume independently — notifications, analytics, fraud checks all react to the same event without touching each other.

Key properties:

Property	Effect
Services never call each other directly	Loose coupling; add consumers without touching producers
Asynchronous delivery	Producers aren’t blocked by slow consumers
Broker owns persistence/retries	Reliable delivery even when consumers are down

The Trade-offs

• Slower than direct calls — an extra network hop and serialization layer.
• Eventual consistency — consumers lag producers; wrong fit when you need strong transactional guarantees.
• Operational complexity — brokers need setup, monitoring, capacity planning.
• Failure handling is your job — retries, duplicates, and ordering must be designed for (see Event-Driven Architecture for idempotency).

When to Reach For It

• Work that doesn’t need an immediate response (emails, thumbnails, analytics).
• Spike absorption (enqueue now, process later).
• Fan-out to multiple independent consumers.

Avoid it for request/response paths that need a synchronous answer.

---

Part of the Messaging series.