Message Queues

Imagine you are building a video platform. When a user uploads a video, the server must create several resolutions, make thumbnails, and check the content. Creating the resolutions alone can take several minutes.

If the upload request waits for all this work, the user stares at a loading screen, and the request may time out before the work is done.

There are two more problems:

• Overload: If 100 users upload at the same time, the server tries to process all 100 videos at once and runs out of CPU and memory.
• Fragile request: If the processing fails halfway, the whole upload request fails, even though the video was uploaded fine.

The root problem is that slow, heavy work is tied to the user's request. A better plan is:

1. Accept the upload quickly.
2. Write down the tasks that need to be done.
3. Let separate workers do them later, on their own.

This is exactly what a message queue is for.

What Is a Message Queue?

A message queue (for example Amazon SQS or RabbitMQ) is a waiting line that sits between two kinds of services:

• Producer: the service that creates work.
• Consumer: the service that does the work.

The producer puts a message in the queue, like "process video 456". The queue keeps it safe until a consumer is free. The consumer takes the message, does the work, and tells the queue "done".

1Upload Service  --- send message --->  Queue  --- deliver --->  Worker
2 (Producer)                         "process video 456"        (Consumer)
3                                                                   |
4                                          <------ acknowledge -----+

The system that stores and hands out the messages is called the broker. It accepts messages, keeps them safe, delivers them, and remembers which ones are finished.

One Message, One Consumer

In a queue, each message goes to only one consumer. If 10 workers read from the same queue, each message is handled by just one of them.

This is different from pub/sub, where every subscriber gets its own copy of each message.

Queue = hand out work. Pub/sub = announce that something happened.

Back to the Video Example

Now the upload service saves the file, drops a message in the queue, and replies immediately. The user sees "Your video is processing", and the heavy work runs in the background.

The Life of a Message

Let us follow one message from start to end.

1Producer          Queue                    Consumer 1        Consumer 2
2   |                |                          |                 |
3   |-- send ------->|                          |                 |
4   |                |-- store safely (disk) -->|                 |
5   |<-- confirm ----|                          |                 |
6   |                |-- deliver -------------->|                 |
7   |                |   (hidden from others)   |                 |
8   |                |<-- acknowledge ----------|                 |
9   |                |   (message removed)      |                 |
10 
11If Consumer 1 crashes before acknowledging:
12   |                |   (timeout ends, message visible again)    |
13   |                |-- deliver again ----------------------->   |

Step 1: The producer sends the message. The queue stores it safely, usually on disk and often copied to more than one machine. Only then does it confirm to the producer. After the confirmation, the producer can move on.

Step 2: A consumer receives it. The queue does not delete the message yet. It only hides it from other consumers. In Amazon SQS this is the visibility timeout. For a set time (say 30 seconds), the message is invisible to others while the first consumer works on it.

Step 3: The consumer acknowledges. When the work is done, the consumer sends an acknowledgment. In SQS, this means deleting the message from the queue.

What If the Consumer Crashes?

The consumer never acknowledged, so the queue still has the message:

• SQS: the visibility timeout runs out, and the message becomes visible again.
• RabbitMQ: the broker sees that the consumer's connection closed and puts the message back.

Another consumer picks it up and tries again. This is how acknowledgments make sure no message is lost.

The Catch: Duplicates

Suppose a consumer finishes the work but crashes before acknowledging. The message is delivered again, and the work runs twice. So consumers must be safe to run more than once. A later chapter on delivery semantics covers how to do this.

Scaling with Competing Consumers

Scaling a queue is easy: add more consumers to the same queue. Each worker takes whatever message is available, so the work is shared. This is called the competing consumers pattern.

1                        +--> Worker 1  (video 101)
2Upload Service --> Queue +--> Worker 2  (video 102)
3                        +--> Worker 3  (video 103)

Scaling on Queue Depth

Queue depth is the number of messages waiting. Many systems use it to scale automatically:

• Queue keeps growing → add more workers.
• Queue is almost empty → remove workers to save money.

More workers help only up to a point. If all workers use the same database or the same outside API, that shared thing becomes the limit, and extra workers just put more pressure on it.

Prefetch

Prefetch means a consumer takes several messages at once instead of one, to save network trips.

The risk: one worker grabs 100 messages while the others sit idle. In RabbitMQ you control this with the prefetch count.

• Long tasks (like video processing): keep prefetch small, so work is spread evenly.
• Short, light tasks: a larger prefetch gives better speed.

Absorbing Traffic Spikes

A queue also protects your system from sudden traffic spikes.

Imagine a flash sale. Orders jump from 100 per second to 5,000 per second for ten minutes. The order service can accept them quickly, but the fulfillment system can handle only 1,000 per second.

1Flash sale             Order Service            Order Queue             Fulfillment
25,000 orders/s  ---->  (accepts fast)  ---->  (backlog grows)  ---->  Workers
3                                                                     steady 1,000/s

• Without a queue: fulfillment is overloaded, requests start failing, and the failures spread back to the order service.
• With a queue: extra orders wait in line. Workers keep a steady 1,000 per second, and the backlog clears once traffic is normal again.

This is called load leveling. It works only because it is fine for fulfillment to happen a few minutes later. If the user needed the result instantly, a queue would not remove that need.

The Limits of Buffering

If messages always arrive faster than consumers can process them, the backlog never stops growing. Users wait longer and longer, and the queue may eventually run out of space. So watch two numbers:

The age of the oldest message is often more useful. A queue with only 100 messages can still be in trouble if the oldest one has waited an hour.

Poison Messages and Dead-Letter Queues

Some messages fail every time, no matter how often you try. Maybe the video file is broken, or the message points to a user account that was deleted.

Each failure puts the message back in the queue, and another worker tries it again. With no limit, this repeats forever, wastes resources, and delays other messages. Such a message is called a poison message.

1Main Queue --> Consumer --> Success --> Message deleted
2                   |
3                   +--> Failure --> Tried fewer than maxReceiveCount times?
4                                          |                    |
5                                         yes                   no
6                                          |                    |
7                                  Back to Main Queue    Dead-Letter Queue (DLQ)
8                                                               |
9                                                  Investigate, fix, then reprocess

The fix is to limit the number of attempts. In Amazon SQS, this setting is maxReceiveCount. When a message has been received that many times without being deleted, SQS moves it to a separate queue called a dead-letter queue (DLQ).

Now the bad message no longer blocks normal work. Engineers can look into the cause, fix it, and reprocess the message later.

Message Ordering

The word "queue" sounds like first in, first out. But in real systems, order is not always kept.

Even if messages are delivered in order, workers run at different speeds:

• Worker 1 takes message 1 but gets stuck on a slow network call.
• Worker 2 takes message 2 and finishes first.

Retries also break order, because a failed message may be retried after newer messages are already done.

Often this is fine. For two different videos, it does not matter which finishes first.

But sometimes order matters. Say message 1 changes the shipping address and message 2 says ship the order. If "ship" runs first, the package goes to the old address.

Ordering Within a Group

The usual fix is to keep order inside a group of related messages, not across the whole queue.

1SQS FIFO Queue
2 
3order-17: update address -> order-17: ship order   --> Worker 1 (in order)
4order-42: update address -> order-42: ship order   --> Worker 2 (in order)

Amazon SQS FIFO queues do this with a message group ID. Messages with the same group ID (for example, the same order) are delivered in order. Different groups run in parallel.

So you get order where it matters, without pushing everything through one worker. Keeping strict order for every message in a busy queue rarely scales.

Keep order only inside the business entity that needs it (like an order ID or account ID), not globally.

Designing Messages

Messages need careful design too.

Keep them small. Queues are built for many small messages, and most have a size limit. For a large file, store it in object storage like Amazon S3 and put only a reference in the message.

For example, a video message should carry the video ID and its location, not the video itself:

1{
2  "messageId": "msg-7f3a91",
3  "type": "TRANSCODE_VIDEO",
4  "version": 1,
5  "videoId": "456",
6  "storagePath": "s3://uploads/videos/456.mp4"
7}

Useful extra fields:

• Message ID: lets consumers detect duplicates.
• Message type: tells consumers what kind of task it is.
• Version: tells consumers how to read the data. You can change the message format later without breaking consumers that still expect the old version.

When to Use Message Queues

Use a queue when a task is slow, heavy, or does not need to finish before the user gets a response. For example:

• Processing videos and images
• Sending emails and notifications
• Generating reports
• Background payment steps
• Handling webhook events

A queue also helps when:

• One service creates work faster than another can handle it.
• A service is unreliable, and you want failed tasks to be retried, not lost.

Do not use a queue when the caller needs an instant answer, such as checking a password at login. A direct request is simpler there.

Summary

A message queue lets a producer hand off work without waiting, while consumers process it at their own speed.

• Acknowledgments stop messages from being lost.
• Competing consumers make scaling easy.
• Buffering absorbs sudden traffic spikes.

To use queues safely, also remember:

• Handle duplicates, because delivery is usually at-least-once.
• Set retry limits and dead-letter queues for poison messages.
• Think about ordering, and use message groups where order matters.
• Keep messages small and versioned.
• Watch queue depth and the age of the oldest message.

Interview Answer: What Is a Message Queue?

A message queue is a buffer between a producer, which creates tasks, and a consumer, which does them. The producer sends a message and moves on, and a consumer processes it later and acknowledges it. If the consumer crashes, the message is delivered again, so nothing is lost. The cost is that duplicates can happen, so consumers must be safe to run twice.

Queues let us scale by adding more consumers, and absorb traffic spikes by letting work wait in line. For messages that keep failing, we set a retry limit and move them to a dead-letter queue. Where order matters, we keep it only within a group, such as one order ID.

We use queues for slow or background work, like video processing or sending emails, and avoid them when the caller needs an immediate answer.