In a message queue, can 2 consumers consume messages of the same type? In C.
Posted by Avi250@reddit | learnprogramming | View on Reddit | 10 comments
Hello, everyone; I have a server and multiple clients, who send their requests to the server. At one point, there could be another server (I'm calling them servers for a lack of a better word) to help the first server out. However, I need the consumers to be in a queue. I know it is agreed upon to have the messages' type be 1 and the server answers back with a message whose type is the client's pid. I also know message queues are destructive.
That said, can I have 2 servers consume type-1 messages? They'd have to race for them, which isn't a problem (at least not in my simulation: think restaurant or ticket offices). However, I need each message to be consumed, and only be consumed once. Even if it is possible, is it a practice that is frowned upon? If yes, why?
I'm writing this in C on linux, if it's necessary information.
Thank you in advance!
igotshadowbaned@reddit
Look up pthreads and mutex locking
esaule@reddit
Yes. This is a very common thing. Consult the manual of whatever you use to figure out how they behave with multiple writers and multiple readers.
You need to make sure that the access to the queue are atomic, that is to say that they can't be interrupted.
JGhostThing@reddit
Many html servers support multiple processes to better deal with requests.
You do have to be careful taking things off the queue. It's best to have separate input and output queues.
Have fun!
dyslechtchitect@reddit
Coming from the world of SaaS it really depends on the queue implementation. Some queues have consumer groups where a message is visible to the group and consumed by one consumer(so having multiple groups would result in multiple consumptions) Kafka works like that. Other solutions like SQS do not have this concept and require fan out through multiple queues usually via something intermediary like SNS. I imagine lower level implementations work similarly.
Justin_Passing_7465@reddit
Another approach is topics-vs-queues. JMS and MQ implementations do this. If you create a "topic", then every consumer gets a copy of every message in that topic. If you instead create a "queue", then the system tries to deliver each message to only one consumer (though that can fail in edge cases because exactly-once semantics are practically impossible).
DrShocker@reddit
Sure, I think stuff like NATS and Kafka both support multiple consumers.
Avi250@reddit (OP)
I don't think I'm allowed to use tools that haven't been covered during our lesson, I'd have to use strictly System V mq. Would that still be ok? Of course, I'd make a new structure for the message
LeeRyman@reddit
What have you tried?
If your clients are receiving on the same queue, there is your example of multiple receivers. The documentation says that...
There is nothing in there to say you can't have multiple receivers asking for the same type, but it also doesn't say how the kernel decides who gets it first, so a safe assumption might be that it is an arbitary choice between processes that have called msgrcv with a common type.
We could get a better answer if we (use the source, Luke)[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/ipc/msg.c#l61], or (RTFM)[https://tldp.org/LDP/tlk/ipc/ipc.html#:\~:text=Reading%20from%20the%20queue,up%20and%20run%20again.] -- If I read that correctly, processes calling rcv get put into a _read queue_, they are going to see arriving messages first-come-first-served, and will remain in that read queue until a message of the requested _msgtyp_ is seen. That aligns with what I remember too.
As for "how" to use them - that is completely up to you to design. You design it to fit your needs and operate within the documented constraints and behaviours.
If I was reviewing work, I would be looking at things that make your application reliable and robust, and avoid leaking resources, like: How you generate and share the keys; How do you handle a client or servers going away (what happens if messages are left in the queue for it); How you handle application lifecycle properly (how do you cleanly and gracefully exit an application but not leave messages unanswered, which may involve another form of IPC); How do you handle queue lifecycle properly (how is it created and destroyed, that might require yet other forms of SysV IPC); How do you handle anticipatable errors (the queue being deleted out from under you, or you not having permission to access it, or it running out of message space, or trying to send or receive too large a message); How do you avoid deadlocks, Etc. The actual receiving and sending is probably one of the easiest parts.
(You might ask is this SysV stuff relevant today. Funnily enough it was only relatively recently that I was maintaining and extending code that used SysV MQs, semops _and_ shmem. Even if you end up using more recent posix queues, message brokers, load balancers and more elaborate application protocols, etc, principles of robust design in distributed computing are still very relevant!)
Evening-Living842@reddit
yeah works fine in practice
HashDefTrueFalse@reddit
Yes. Having more than one message type isn't a given. There are many multi-consumer (and multi-producer) message queues that only deal with one type of message. If you're writing it then you can make it work however you like.