I have created two separate application for publisher and consumer that send and receives message from the common queue. I have started consumer application first and the exchange was not declared yet (It has been declared and bind to queue on publisher) that's why I got the error.
So my question is,
Is it a good idea to declare and bind exchange to queue on consumer?
In case of declaring exchange in consumer, should consumer know about the exchange properties and exchange type. Here in my case, consumer only knows the exchange,queue,route to receive message from the particular queue.
First it is important to note that starting a consumer and consuming from an existing queue should not give you any error even it the queue is not bound to any exchange. It is not necessary for a queue to be bound to an exchange, it can exist on its own.
To answer your questions: This depends on your use case. It may be OK for the consumer to create the queue, create the exchange and then bind the queue to the exchange. This allows the consumer to have control which messages are routed to the queue and can be consumed by him. If the consumer is the party that should execute this control, this is fine. But if the use case indicates that another party but the consumer shall control the routing, this other party shall create the exchange and the binding.
Consider a topology where there are only simple exchanges bound to queues. In such a topology there would be only bindings from exchanges to queues but none from exchanges to exchanges. Such a topology can be created by the consumer.
But consider a different topology with two levels of exchanges. The exchanges on the lower level are bound to queues, this is similar to the first topology. But above this lower level there is a higher level of exchanges which are only bound to exchanges on the lower level. The exchanges on the higer level distribute messages based on rules that are not related to concrete consumers. In fact, the two levels of exchanges could exist without any queues and consumers. The creation of the exchanges and bindings in this topology can not be done by a consumer.
A consumer could become a part of the second topology by declaring a queue for himself, binding this queue to the exchanges on the lower level he is interested in, and consume from the queue. The consumer would not create any exchanges, he would just bind his queue to them.
So to sum up: In trivial scenarios, it does not matter who declares exchanges, queues and bindings, as long as everything is done in the right order. But on more complex scenarios, the responsibility should by spread between the RabbitMQ admin, produces and consumers.
Related
I've created an Azure Function App which is triggered from Azure Service Bus Queues. The Service Bus has two queues in it and there is a function with a trigger for each of the queues. The Function App is developed using C# in Visual Studio and uses package deployment publishing.
What I would like to do is be able to indicate that one function/trigger should be processed before the other if they both have messages waiting. (They both do basically the same thing but one queue is for handling messages with a higher priority since queues are only FIFO.)
I have read that functions are processed in alphabetical order but that doesn't feel like something to rely on really.
Is there any way to explicitly indicate a priority (or even a scale-out preference) for one function/trigger over another?
(They both do basically the same thing but one queue is for handling messages with a higher priority since queues are only FIFO.)
The above scenario looks like competing consumers which has a dedicated design pattern known as competing consumers pattern where it contains the limitation with the messaging order in this pattern.
Consumer service instances may receive messages in any order, and this order need not correspond to the order in which the messages were created.
So unfortunately, it's not possible to prioritize one function over another function listening to the Service Bus queue.
You can control the activity function & orchestrator but not the starter function using the Azure Durable Functions.
Microsoft Azure Service Bus Queues can implement guaranteed first-in-first-out ordering of messages by using message sessions. For more information, see Messaging Patterns Using Sessions.
I am using MassTransit along with Azure Service Bus.
All consumers APIs automatically create queues in ASB, but if a consumer is removed the queue stays.
Even worse, if the message type that was used for the removed consumer still exists and is being used throughout the system (for other consumers), the old queue still accepts and hoards messages.
An example would be:
Consumer 'A' accepts messages of type 'mA'.
Consumer 'B' also accepts messages of type 'mA'.
For both of these, a queue is created in ASB (Queue A and Queue B).
Now because of some changes, Consumer 'A' is no longer needed and is deleted from the API - but if message 'mA' is published, both Queue A and Queue B will still receive it.
An obvious solution would be to just have a pretty low Auto-delete setting, but this will only remove the unneeded messages, not the queues themselves.
In NServicebus 7 you can set concurrency that means you can decide how many messages in queue your software can process in parallel.
This can be done at NserviceBus Endpoint level.
I have few doubts about this concept:
the concurrency is per queue not per message Type? Right?
If I use satellites which means I’ll have N different queues (for example: one per message Type), the concurrency will still be per queue?
For example:
I have configured 1 endpoint (so 1 queue) and setted to 10 the concurrency level. I manage 5 different commands (handlers). All the commands are stored in the same queue, mixed. In this case the endpoint is able to take 10 commands per time from the queue without considering the type, correct?
In a second scenario i have 5 satellites which manage the 5 message types, 1 dedicated queue per type. In this case each satellite is able to take 10 messages per time from its queue?
Satellites are an advanced feature for the raw processing of messages without all the benefits of the NServiceBus message processing pipeline. It's not normal to use them—they're most often used when implementing a message transport. For example, the RabbitMQ transport uses a Satellite for a feature that makes an endpoint instance individually addressable, so you have a QueueName queue plus a QueueName-InstanceName queue on the broker, so that another endpoint can do context.Reply() and have the reply go to the specific server that sent the original command. In any case, each satellite manages its concurrency separately, as it's a more low-level construct.
So yes, the concurrency through the main queue is for the endpoint instance, not per message type, because there's a 1:1 relationship between endpoint and queue, and you can't selectively pull messages off the queue by type.
As a result, the endpoint is your unit of scalability, both scaling up (by increasing the concurrency) or out (by adding more endpoint instances on different servers).
This means you should be careful about what message types you process in the same endpoint. They should generally have the same SLA. You wouldn't want a bunch of messages that take 50ms to process held up behind a glut of messages that process for 20 seconds.
Some people will take this to an extreme and go with one endpoint per message type. This level of complexity is usually not necessary, but it does give you the ultimate control over scalability for every single message type.
Is possible to use topic exchange as true event notification system?
I've created topic exchange on given exchange named as Cherry. I've got one publisher at routing key cherry.user.created and many consumers with same routing key, but when I publish an event only one of consumers consume an event. I thought that topic can be used as "real event broadcasting" - every consumer gets notified when given event happened, but right now only one consumer consume an event and other consumers do not know about created event...
To clarify my comment about queues. In rabbitmq, if multiple consumers use the same queue - message delivered to that queue is always dispatched in round-robin manner, no matter what. So when you subscribe to topic exchange, best way is to declare new queue for each consumer (with any name, or better random generated by rabbit itself) and use target routing key (cherry.user.created) to bind those queues to exchange.
I successfully setup a Topic Exchange, and I'm able to deliver messages to several consumers at once.
I'd also like to deliver messages to competing consumers and keep using topic exchanges. I read that using the same queue name enables consumers to compete for messages. I might be mistaken, however, since I cannot make it work.
Setup for several listeners to the same topic:
Declare topic exchange
For each listener, declare a new queue with an autogenerated name
Bind this queue to the above exchange with a given topic routing key
How to setup competing consumers to the same topic?
Is it even possible with Topic Exchanges?
Thanks.
Let's review a couple of points first.
First, remember that in RabbitMQ you always consume from queues. Exchanges are just your portals and you cannot directly consume from them.
Second, Topic exchanges allow for binding the queues with routing key "patterns". Therefore, the term topic is valid in the context of "Topic Exchanges".
Now this is what I understand from your question:
Multiple consumers/same routing key:
This is where you want multiple consumers to all consume the messages with the same routing key (or same routing key patterns in the case of Topic Exchanges). This is in fact doable. Just do this:
Declare your Topic Exchange
Declare a queue with some name
Bind that queue to your topic with your desired routing key pattern
Create multiple consumers and have them listen to that same queue.
What will happen is that RabbitMQ is going to load balance to your consumers in a round robin matter. This means all consumers will consume from the same queue. But remember that in this scenario it is possible that a single message is delivered more than once in theory.
What you were doing was to create multiple queues and have one consumer per queue. This means that every message coming to the exchange would be duplicated across all queues. The end result would be that a message gets processed multiple time.
I solved this by using exchange-to-exchange bindings.
The outer exchange is a topic exchange.
The inner exchange is a fanout exchange bound to a client-named queue.
The outer exchange is bound to the inner exchange with a routing key that includes a wildcard.