Message Queue vs Task Queue vs Message Broker: why are these always mixed up?
Posted by Civil_Station_1164@reddit | programming | View on Reddit | 13 comments
Title: Message Queue vs Task Queue vs Message Broker: why are these always mixed up?
While working with Celery, Redis, and RabbitMQ, I kept seeing people use message queue, task queue, and message broker interchangeably.
After looking into the documentation and real implementations, here’s how I understand it:
Message Queue: just moves messages (one consumer per message).
Message Broker: manages queues, routes, retries, and protocols.
Task Queue: executes actual jobs using workers.
They’re not alternatives; they work together in production systems.
One interesting thing I noticed is that a lot of confusion comes from tools like Redis, which can act as both a simple queue and a broker-like system, and Celery, which abstracts everything.
I’m curious how others think about this. Do you keep these concepts separate in your architecture or treat them more loosely?
I also wrote a deeper breakdown with examples (Celery, RabbitMQ, SQS) if anyone’s interested.
CypherNetSecurity@reddit
They tend to look interchangeable at a conceptual level, but the differences really show up once messages start flowing through the system under real workloads and failure conditions.
my_beer@reddit
I think there is a more fundamental breakdown missing here, messages really fall into two categories, commands and events.
Commands are reqests for a piece of work to be done, these belong on queues (RabbitMQ, SQS etc). Here you are looking for the fan out, retry etc capabilites of a queue.
Events are reports that something has happened (you should always be able to describe them as past tense verb phrases). These belong on streaming systems (Kafka, Kinesis etc) and you are looking at a pub/sub pattern or similar.
Civil_Station_1164@reddit (OP)
Thanks for suggestion. I will definitely work on this.
BuriedStPatrick@reddit
A lot of libraries operate under the assumption that the message producer is also the broker.
If I'm publishing a message to, say, a database to be consumed out-of-process, then I've assumed the broker role as well as the producer. Because I've already made the decision on how this message should be distributed in the system.
I think a lot of these libraries want to make it easier for us and just put in the business logic and let it handle the rest. Absolutely fine, but abstraction does lead to confusion sometimes.
pee_wee__herman@reddit
All I know is that RabbitMQ is the shit for these kinds of things
krypticus@reddit
Is THE shit, or IS shit?
lugh_longarm@reddit
Kafka is the best illustration of why this gets muddled. It markets itself as a distributed log, but teams use it as a broker and a queue simultaneously - and it behaves differently enough that the mismatch causes real problems. Consumer group semantics vs competing consumers, log compaction vs TTL, retention-based replay vs DLQ. People hit these walls mid-incident.
The Redis + Celery confusion is similar. Raw Redis as a queue gives you none of the broker guarantees by default - no real ACK, no DLQ, tasks vanish on crash unless you've tuned visibility_timeout carefully. That's why a lot of teams end up accidentally building broker-like behaviour on top of Redis and then wondering why it's fragile.
The distinction matters most at the failure boundary: what guarantee does each layer make about delivery, routing, and execution? Once you frame it that way the layers stop feeling interchangeable.
Somepotato@reddit
I mean...redis streams operates as a broker just fine
Civil_Station_1164@reddit (OP)
You understood it perfectly but I would like to correct you at one point.
Kafka is not a Task Queue service. A task queue follows the producer consumer design pattern where you can make only one to one connections which actually means that only a single consumer or worker can execute task from the queue. A single task cannot be sent to multiple workers.
On the other hand Kafka which is a real time streaming service works on the principle of Publisher Subscriber Pattern where you can make one to many, many to many and many to one connection which means you can send the same task from the buffer to multiple Consumers (we call them subscribers here) and instead of using queues which can only send a single task to only single worker, we use topics here. Topics can send single task to multiple subscribers. Subscribers need to subscribe topics.
If you want to learn this in detail. Read this blog : https://medium.com/@yashvaishnav1404/producer-consumer-and-pub-sub-when-each-one-makes-sense-56d31d7a4590
SaxAppeal@reddit
Have you ever used Kafka?
lupercalpainting@reddit
Pub/Sub vs Producer/Consumer.
The NYT is one publisher with many subscribers.
The chef produces one sandwich which is eaten by one consumer.
You CAN do producer/consumer with Kafka, but it’s by default set up for pub/sub.
Contrast that with SQS which is set up for producer/consumer but you can do producer/consumer if you add SNS fan out.
granadesnhorseshoes@reddit
I think its a good example of the overall accidental complexity in a lot of modern systems.
What exactly is the functional difference of "a task queue" vs "a message queue"? If a "task queue" is a stand alone service and independent workers simply check the queue for their next task, then its functionally just a "message queue" for task workers isn't it?
So we end up with a half a dozen different implementations of the same basic overall concepts all with their own semantics that can be fudged around and used interchangeably even if its not optimal or down right crazy and confusing for a given use case.
posts_saver@reddit
have same words in their names and are used together