Message Queues: A Simple Guide with Analogies (2024)

This is surprisingly basic knowledge for ending up on the front page.

It’s a good intro, but I’d love to read more about when to know it’s time to replace my synchronous inter service http requests with a queue. What metrics should I consider and what are the trade offs. I’ve learned some answers to this question over time, but these guys are theoretically message queue experts. I’d love to learn about more things to look out for.

There are also different types of queues/exchanges and this is critical depending on the types of consumer or consumers you have. Should I use direct, fan out, etc?

The next interesting question is when should I use a stream instead of a queue, which RabbitMQ also supports.

My advice, having just migrated a set of message queues and streams from AWS(AvtiveMQ) to RabbitMQ is think long and hard before you add one. They become a black box of sorts and are way harder to debug than simple HTTP requests.

Also, as others have pointed out, there are other important use cases for queues which come way before microservice comms. Async processing to free up servers is one. I’m surprised none of these were mentioned.

I’ve been thinking that defaulting to durable execution over lower‑level primitives like queues makes sense a lot of the time, what do you think?

A lot of the "simple queue" use cases end up needing extra machinery like a transactional‑outbox pattern just to be reliable. Durable‑execution frameworks (DBOS/Temporal/etc.) give you retries, state, and consistency out of the box. Patterns like Sagas also tend to get stitched together on top of queues, but a DE workflow gives you the same guarantees with far less complexity.

The main tradeoff I can think of is latency: DE engines add overhead, so for very high throughput, huge fan‑out, or ultra‑low‑latency pipelines, a bare‑bones queue + custom consumers might still be better.

Curious where others draw the line between the two.

While queues definitely play an important role in microservices architecture, I think it’s worth clarifying that they’re not unique to it. A queue can fit perfectly in a monolith depending on the use case. I regularly use queues for handling critical operations that might require retrying, for having better visibility into failed jobs, ensuring FIFO guarantees, and more. Queues are such a useful tool for building any resilient architecture that framing them as primarily a microservices concern might cause unnecessary confusion.

After spending most of my career hacking on these systems, I feel like queues very quickly become a hammer and every entity quickly becomes a nail.

Just because you can keep two systems in complete sync doesn't mean you should. If you ever find yourself with more-or-less identical tables in two services you may have gone too far.

Eventually you find yourself backfilling downstream services due to minor domain or business logic changes and scaling is a problem again.

this is not really a good guide to Message Queues, it's really only talking about them in context of one use of them. It doesn't really talk about the message queue at all and the basic differences between various message queue implementations. Your local AI chatbot is going to give you a much better overview, just take the title "Message Queues: A Simple Guide with Analogies" and it does a much better job