Session 27: Absorbing Load Without Waiting: Message Queue, Backpressure, & DLQ

1. Imagine If

At a fast-food restaurant, the cashier takes a burger order and money from a customer. - Synchronous Scenario (Bad): The cashier leaves the counter, walks to the kitchen, fries the patty for 10 minutes, toasts the bun, wraps the burger, and then hands it to the customer. During those 10 minutes, the line at the counter snakes all the way out to the street and the whole shop grinds to a halt. - Asynchronous Scenario (Message Queue): The cashier prints a queue-number receipt in 2 seconds, hands it to the customer, and clips the order ticket onto the kitchen's wire rail. The customer steps aside calmly. The cooks take the tickets one by one at their own working pace. The cashier can serve 500 new customers without ever being held back by the speed of the stove!

2. What Actually Happens

In modern distributed architecture, separating the Critical Path (Hot Path) from the Asynchronous Path (Background Job) is the key to performance:

1. Synchronous vs Asynchronous Decoupling: - Hot Path (Synchronous): Only process the vital things that are absolutely needed right now (e.g. verify balance & record the transaction \rightarrow 20 ms). Return a 202 Accepted or 200 OK status to the user immediately. - Background Job (Asynchronous): Send an OrderCreated event to the Message Broker (Kafka / RabbitMQ).

2. Two Main Types of Broker: - Queue-Based (RabbitMQ): Messages are put into a queue, picked up by a worker, and deleted once acknowledged (ACK). A great fit for task distribution and complex routing. - Log-Based (Apache Kafka): Events are stored as an ordered log sequence (append-only commit log) that persists on disk. Many consumer groups can re-read (replay) data from a given offset without deleting events. A good fit for massive-scale event streams.

3. Consumer Lag (The Most Honest Metric): -

   \text{Consumer Lag} = \text{Latest Produced Offset} - \text{Current Consumer Offset}
   - Shows how many messages are piling up in the broker's queue. If Lag keeps rising under high load, that's a red alert that the workers are short on capacity or stuck!

4. Backpressure & Dead Letter Queue (DLQ): - Backpressure: An automatic braking mechanism that tells the sending system (producer) to slow down its sending rate when the receiver (consumer) starts getting overwhelmed. - Poison Pill: A message with a broken format (malformed payload) that makes the consumer application crash with an exception every time it tries to process it. Without handling, a single poison message will make the worker restart endlessly (infinite crash loop). - Dead Letter Queue (DLQ): If a message fails to be processed after N attempts (e.g. 3 retries), it is automatically moved to "quarantine" (the DLQ) so the main queue keeps flowing smoothly, and the engineering team can inspect the poisonous message manually.

3. The Official Name

• Message Broker: Intermediary software that enables asynchronous communication between services through messages.
• Consumer Lag: The difference in message count between what has been produced and what the consumer has finished processing.
• Backpressure: A resistance signal from downstream to upstream to regulate the speed of the data flow.
• Dead Letter Queue (DLQ): A dedicated secondary queue to hold messages that have failed processing repeatedly.
• At-Least-Once Delivery: A message delivery guarantee in which a message is ensured to arrive at least once (requiring idempotency on the consumer side).

4. In Our World

• Uber / Gojek Receipt Generation: Trip receipts and reward point calculations are processed via Kafka/SQS in the background a few seconds after the trip ends.
• Tokopedia / Shopee Flash Sale: Push notifications and order confirmation emails are streamed through a RabbitMQ/Kafka worker pool.

5. The Performance Tester's Lens

Crucial metrics and tests: 1. Producer Ingress Throughput: Test the publisher's capacity to produce 50,000 events/second without hitting a disk bottleneck. 2. Consumer Lag Drain Rate: Stop all consumers for 10 minutes while traffic runs at 5,000 QPS, then turn them back on. Measure how many seconds the consumer pool needs to empty the queue (drain lag). 3. Poison Pill Isolation: Inject 10 invalid payloads (null pointer triggers) in the middle of 100,000 normal messages. Make sure those messages land in the DLQ without hurting worker throughput.

6. Question for the Next Round

Once the backend is resilient thanks to message queues, how do we make sure no hacker, scraper bot, or DDoS attack drains our bandwidth and server resources at the outermost gate?

The answer is in Session 28: Throttling the Flow: Rate Limiting, Leaky Bucket, & Token Bucket.