I successfully setup a Topic Exchange, and I'm able to deliver messages to several consumers at once.
I'd also like to deliver messages to competing consumers and keep using topic exchanges. I read that using the same queue name enables consumers to compete for messages. I might be mistaken, however, since I cannot make it work.
Setup for several listeners to the same topic:
Declare topic exchange
For each listener, declare a new queue with an autogenerated name
Bind this queue to the above exchange with a given topic routing key
How to setup competing consumers to the same topic?
Is it even possible with Topic Exchanges?
Thanks.
Let's review a couple of points first.
First, remember that in RabbitMQ you always consume from queues. Exchanges are just your portals and you cannot directly consume from them.
Second, Topic exchanges allow for binding the queues with routing key "patterns". Therefore, the term topic is valid in the context of "Topic Exchanges".
Now this is what I understand from your question:
Multiple consumers/same routing key:
This is where you want multiple consumers to all consume the messages with the same routing key (or same routing key patterns in the case of Topic Exchanges). This is in fact doable. Just do this:
Declare your Topic Exchange
Declare a queue with some name
Bind that queue to your topic with your desired routing key pattern
Create multiple consumers and have them listen to that same queue.
What will happen is that RabbitMQ is going to load balance to your consumers in a round robin matter. This means all consumers will consume from the same queue. But remember that in this scenario it is possible that a single message is delivered more than once in theory.
What you were doing was to create multiple queues and have one consumer per queue. This means that every message coming to the exchange would be duplicated across all queues. The end result would be that a message gets processed multiple time.
I solved this by using exchange-to-exchange bindings.
The outer exchange is a topic exchange.
The inner exchange is a fanout exchange bound to a client-named queue.
The outer exchange is bound to the inner exchange with a routing key that includes a wildcard.
Related
I have created two separate application for publisher and consumer that send and receives message from the common queue. I have started consumer application first and the exchange was not declared yet (It has been declared and bind to queue on publisher) that's why I got the error.
So my question is,
Is it a good idea to declare and bind exchange to queue on consumer?
In case of declaring exchange in consumer, should consumer know about the exchange properties and exchange type. Here in my case, consumer only knows the exchange,queue,route to receive message from the particular queue.
First it is important to note that starting a consumer and consuming from an existing queue should not give you any error even it the queue is not bound to any exchange. It is not necessary for a queue to be bound to an exchange, it can exist on its own.
To answer your questions: This depends on your use case. It may be OK for the consumer to create the queue, create the exchange and then bind the queue to the exchange. This allows the consumer to have control which messages are routed to the queue and can be consumed by him. If the consumer is the party that should execute this control, this is fine. But if the use case indicates that another party but the consumer shall control the routing, this other party shall create the exchange and the binding.
Consider a topology where there are only simple exchanges bound to queues. In such a topology there would be only bindings from exchanges to queues but none from exchanges to exchanges. Such a topology can be created by the consumer.
But consider a different topology with two levels of exchanges. The exchanges on the lower level are bound to queues, this is similar to the first topology. But above this lower level there is a higher level of exchanges which are only bound to exchanges on the lower level. The exchanges on the higer level distribute messages based on rules that are not related to concrete consumers. In fact, the two levels of exchanges could exist without any queues and consumers. The creation of the exchanges and bindings in this topology can not be done by a consumer.
A consumer could become a part of the second topology by declaring a queue for himself, binding this queue to the exchanges on the lower level he is interested in, and consume from the queue. The consumer would not create any exchanges, he would just bind his queue to them.
So to sum up: In trivial scenarios, it does not matter who declares exchanges, queues and bindings, as long as everything is done in the right order. But on more complex scenarios, the responsibility should by spread between the RabbitMQ admin, produces and consumers.
I am new to Azure Service Bus and would like to know if I can multiple subscribers to a queue or topic? In rabbit MQ I can have multiple subscribers to 1 publisher.
What I am trying to do is, I am using CQRS and when certain commands come into the system when the event is handled I want to push them into a message queue.
I want 2 subscribers to be able to get the messages from that queue, one for me to process internally. another one for process and send externally.
I am new to Azure Service Bus and would like to know if I can multiple
subscribers to a queue or topic?
Yes. This is possible with Azure Service Bus Topics. There can be multiple subscribers to a message sent to a topic. From this link:
In contrast to queues, in which each message is processed by a single
consumer, topics and subscriptions provide a one-to-many form of
communication, in a publish/subscribe pattern. Useful for scaling to
very large numbers of recipients, each published message is made
available to each subscription registered with the topic. Messages are
sent to a topic and delivered to one or more associated subscriptions,
depending on filter rules that can be set on a per-subscription basis.
The way it works is that you create a topic and then create multiple subscriptions in that topic. In each subscription, you can define message filtering rules. When a message is sent to a topic, Azure Service Bus matches that message against the filtering rules in each subscription and if a matching rule is found, then the message is sent to that subscription.
I wanted to add something to the discussion: from the following article it seems that each listener would have to create a different subscriber in azure before listening to the same topic. Basically a topic will have multiple subscribers (every subscription with a single listener) and so the service bus will know exactly how many subscribers will need to listen to the message before it can be completed.
https://medium.com/awesome-azure/azure-difference-between-azure-service-bus-queues-and-topics-comparison-azure-servicebus-queue-vs-topic-4cc97770b65
Is possible to use topic exchange as true event notification system?
I've created topic exchange on given exchange named as Cherry. I've got one publisher at routing key cherry.user.created and many consumers with same routing key, but when I publish an event only one of consumers consume an event. I thought that topic can be used as "real event broadcasting" - every consumer gets notified when given event happened, but right now only one consumer consume an event and other consumers do not know about created event...
To clarify my comment about queues. In rabbitmq, if multiple consumers use the same queue - message delivered to that queue is always dispatched in round-robin manner, no matter what. So when you subscribe to topic exchange, best way is to declare new queue for each consumer (with any name, or better random generated by rabbit itself) and use target routing key (cherry.user.created) to bind those queues to exchange.
I am evaluating using RabbitMQ as message queue/message bus and have been looking at the example tutorials on the RabbitMQ page.
I am looking for a specific scenario not covered by the tutorials and I am not sure if and how it would be possible to do via RabbitMQ.
The setup:
Let's assume I got a service, let's call it "purchase orders" and I have to other services called "logistics" and "accounting".
When an order is sent, I want to send it as a message via RabbitMQ.
There 2 "account" and 3 "logistic" services
What would be the correct way to ensure that "account" and "logistic" will process the message only once? Using pub/sub will cause the messages to be processed twice (account) or trice (logistics) if i understand it correctly.
With work queues and prefetch=1 it would assure that only one gets it, but I have 2 services and want each type of service to get one.
Is there a way to combine both and have a work queues for each of the service, without sending 2 separate events/messages to two different exchanges?
Using pub/sub will cause the messages to be processed twice (account) or trice (logistics) if i understand it correctly.
you probably have 1 queue per worker, based on your description, and you are routing the message to all worker queues. therefore, each worker gets a copy of the message, because you routed the message to all of the queues.
what you want is a single "account" queue and a single "logistic" queue. you will have multiple account services reading from the single account queue; same for the logistic service / queue.
setting prefetch=1 is important as well. this prevents you from reading too many messages in to a single worker, at once.
Is there a way to combine both and have a work queues for each of the service, without sending 2 separate events/messages to two different exchanges?
yes - don't use a fanout exchange. use a topic or direct exchange, and use multiple routing keys to route a single message to both the account and logistics queues.
What would be the correct way to ensure that "account" and "logistic" will process the message only once?
there is no way to guarantee this, 100%. at some point, even with a proper setup like I've described, you will have a network failure or a worker crash or some other problem and a message will get processed twice. you have to account for this in you design, using some form of idempotence in your message processing.
hope that helps!
I have Created a RabbitMQ Producer and a RabbitMQ Consumer....
suppose my producer produces 10 messages. How can i get a particular message from those 10 messages.
I want to know how can i uniquely identify a message and read that or consume that message.
There are several ways to do this, but the one I use most is to use a routing key that is unique to the type of message. Consumers, then, bind to that exchange using a specific routing key, which causes messages to go only to those consumers.
If you can avoid it, you should never just dump messages into a single queue and let the consumers sort them out. The routing keys and exchanges are powerful tools made specifically for routing messages. You should leverage that.
I have an example that shows how to do a topic queue in C# which appears to be what your looking for RabbitMQ Tutorial I also have one that shows how to use the EventingBasicConsumer to avoid blocking when getting messages RabbitMQ EventingBasicConsumer