I am not able to figure out on how to specify the Exchange and Queue in my GetSendEndpoint()) task when sending / publishing messages?
As per MassTransit documentation https://masstransit-project.com/usage/producers.html#send you can specify the exchange and queue like
GetSendEndpoint(new Uri("queue:input-queue"))
However, I can only do one or the other?
Is there an alternative way of sending with exchange and queue specified?
I am doing this in Asp.Net Core so here are my configuration:
Startup.cs
public void ConfigureServices(IServiceCollection services)
{
services.AddMassTransit();
services.AddSingleton(p => Bus.Factory.CreateUsingRabbitMq(cfg =>
{
cfg.Host("rabbitmq://localhost", h =>
{
h.Username("admin");
h.Password("admin");
});
}));
services.AddSingleton<IBus>(p => p.GetRequiredService<IBusControl>());
services.AddSingleton<IHostedService, BusService>();
}
And this is how send the message
var endpoint = await _bus.GetSendEndpoint(new Uri(queue:Test.Queue));
await endpoint.Send(new Message()
{
Text = "This is a test message"
});
As you can see I can only specify the queue name.
If you specify an exchange, only the exchange is declared on the broker. The message will be sent directly to the exchange.
"exchange:your-exchange-name"
If you specify a queue, the queue along with an exchange of the same name will be declared and the exchange will be bound to the queue. Messages will be delivered to the exchange of the same name, which will deliver them to the queue.
"queue:your-queue-name"
If you want a different exchange and queue name, you can specify both using:
"exchange:your-exchange-name?bind=true&queue=your-queue-name"
Or you could simplify, but it's a little confusing with two queues:
"queue:your-exchange-name&queue=your-queue-name"
Reading Chris's response in here Mass Transit : No consumer
It seems like Exchanges are created by MassTransit when publishing messages, based on the message types. Publishing does not create any queues. Queues are where messages are stored for delivery to consumers.
and Queues are created when receive endpoints are added to a bus. For the consumers, handlers, and sagas added to a receive endpoint, the exchanges are created and bound so that messages published to the exchanges are received by the receive endpoint (via the queue).
So if my publisher doesn't have a receive endpoint defined then any messages I send will be lost as there will be no queues or binding?
Further reading on here https://groups.google.com/forum/#!topic/masstransit-discuss/oVzZkg1os9o seems to further confirm this.
So based on the above link in order to achieve what I want i.e. to create the exchange and bind it to a queue I will need to specify it in the Uri as such
var sendEndpoint = bus.GetSendEndpoint(new Uri("rabbitmq://localhost/vhost1/exchange1?bind=true&queue=queue1"));
where exchange1 is the Exchange, queue1 is the Queue and bind=true would bind the queue to the exchange.
If sticking to the original MT design a Consumers needs to be running before to setup the exchanges and queues before a Producer can start publishing? This seems to give less flexibility to the Publisher?
Related
Someone put a message in RabbitMq (for example). The message is not in the Masstransit "envelop" format, just json, or maybe something else.
How to get this message in Masstransit consumer as byte[], or string?
I understand that Masstransit has its own message format, but there is a need to receive a raw message. I don’t really like the option when Masstransit and RabbitMq client are used in one service, because these are already two connections, and different message processing logic.
On the receive endpoint where raw JSON messages are to be consumed, you can configure a default deserializer as shown:
endpointConfigurator.UseRawJsonDeserializer(isDefault: true);
I have a web application that publishes messages to a topic then several Windows services that subscribe to those topics, some with multiple instances. If the services are running when the messages are published everything works correctly but if they are not then the messages are retained on the queue(s) subscribing to that topic but aren't read when the services start back up.
The desired behavior-
When a message is published to the topic string MyTopic, it is read
from the MyTopicQueue only once. I use some wildcard topics so each message is sent to multiple queues, but multiple instances of a services subscribe to the same topic string and each message should be read by only of those instances
If the subscribers to the MyTopic topic aren't online when the message is published then the messages are retained on MyTopicQueue.
When the Windows services subscribing
to a particularly topic come back on line each retained message is
read from MyTopicQueue by only a single subscriber.
I've found some [typically for IBM] spotty documentation about the MQSUBRQ and MQSO_PUBLICATIONS_ON_REQUEST options but I'm not sure how I should set them. Can someone please help figure out what I need to do to get my desired behavior? [Other than switching back to RabbitMQ which I can't do though I'd prefer it.]
My options:
private readonly int _openOptions = MQC.MQSO_CREATE | MQC.MQSO_FAIL_IF_QUIESCING | MQC.MQSO_MANAGED;
private readonly MQGetMessageOptions _messageOptions = new MQGetMessageOptions()
Code to open the Topic:
_topic = _queueManager.AccessTopic(_settings.TopicString, null,
MQC.MQTOPIC_OPEN_AS_SUBSCRIPTION, _openOptions);
The line of code that reads from the topic (taken from a loop):
_topic.Get(mqMessage, _messageOptions);
If you want the messages to accumulate while you are not connected you need to make the subscription durable by adding MQC.MQSO_DURABLE. In order to be able to resume an existing subscription add MQC.MQSO_RESUME in addition to MQC.MQSO_CREATE.
Be careful with terminology, what you are describing as retained messages is a durable subscription.
Retained publications are something else were MQ can retain one most recently published message on each topic and this message will be retrieved by new subscribers by default unless they use MQSO_NEW_PUBLICATIONS_ONLY to skip receiving the retained publication.
MQSO_PUBLICATIONS_ON_REQUEST allows a subscriber to only receive retained publications on request, it will not receive non-retained publications.
If you want multiple consumers to work together on a single subscription you have two options:
Look at shared subscribers in XMS.NET, look at the CLONESUPP property.
Create a one time durable subscription to a queue on the topics you want consumed, then have your consumers directly consume from the queue not a topic.
Full disclosure: I am new to the Service Fabric development. Here is my situation. We have Service Fabric Cluster. We deployed stateful service there. Service has designated Service Bus queue, which it listens to. So all service instances on all nodes in the cluster listening to the same Service Bus queue. Each service instance register OnMessage callback with the Service Bus queue to process message like this:
QueueClient Queue = QueueClient.CreateFromConnectionString(
GetServicebusConnectionString(),
ConfigData.SERVICE_QUEUE_NAME);
if (Queue != null)
{
var options = new OnMessageOptions();
options.AutoComplete = false;
Queue.OnMessage((receivedMessage) =>
ProcessMessage(receivedMessage), options);
}
Now, according to messages in the log it looks like all service instances pick up message, that has been placed in the queue simultaneously. Which does not constitute a good thing.
Question is:
Is it possible to use Service Bus queue in a way when each message from the queue would be picked up only by one service instance?
The default receive mode of queue client is PeekLock, and you set AutoComplete property to false, which will not automatically delete message after the client has received the message. After the lock expires, the message will become available again, and other service instances could receive and process it again.
Is it possible to use Service Bus queue in a way when each message from the queue would be picked up only by one service instance?
You could set AutoComplete property to true, or call Complete method after the client has received and processed the message.
Queue.OnMessage((receivedMessage) =>
{ ProcessMessage(receivedMessage); receivedMessage.Complete(); }, options);
Using the code as shown here.. I was able to create a web app that every 30 seconds sent data to client using System.Threading.Timer.
I was able to add some code which received data from a service bus queue using Messaging factory and Messaging receiver and based on that sent data to signalR client instead of hard-coding as in the mentioned example.
Now my real application gets data from 50 such queue..
Theoretically, I could create 50 timer objects which would call 50 different methods which in turn would call service bus queue.
I would sincerely appreciate if someone could suggest the right way to achieve my goal..
Thanks
The message pump pattern seems like it would be a good fit for this application. You create a separate client for each queue and configure each one to automatically listen for messages in its queue and process them as they come in.
foreach (var queueName in queueNames){
var queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueClient.OnMessage(message =>
{
// Do work here
Console.Out.WriteLine(string.Format("Recieved message {0} on queue {1}", message.MessageId, queueName));
});
}
How does the poison-message handling work for Azure WebJobs SDK's ServiceBusTrigger ? I am looking to push the service bus queue messages that have been dequeued more than 'x' times to a different ServiceBus (or) Storage queue
The online documentation here and here and SDK Samples from here does not have examples on how the poison message handling works for ServiceBusTrigger. Is this work in progress?
I tried implementing a custom poison message handling using dequeueCount parameter but it doesn't look that it is supported for ServiceBusTriggers as I was getting a runtime exception {"Cannot bind parameter 'dequeueCount' when using this trigger."}
public static void ProcessMessage([ServiceBusTrigger(topicName: "abc", subscriptionName: "abc.gdp")] NotificationMessage message,
[Blob("rox/{PayloadId}", FileAccess.Read)] Stream blobInput, Int32 dequeueCount)
{
throw new ArgumentNullException();
}
While you cannot get the dequeueCount property for ServiceBus messages, you can always bind to BrokeredMessage instead of NotificationMessage and get the property from it.
It looks like WebJobs handles this internally at the moment.
Reference: How to use Azure Service Bus with the WebJobs SDK
Specific section:
How ServicebusTrigger works
The SDK receives a message in PeekLock mode and calls Complete on the
message if the function finishes successfully, or calls Abandon if the
function fails. If the function runs longer than the PeekLock timeout,
the lock is automatically renewed.
Service Bus does its own poison queue handling, so that is neither
controlled by, nor configurable in, the WebJobs SDK.
Additional Reference
Poison message handling can't be controlled or configured in Azure Functions. Service Bus handles poison messages itself.
To add to Brendan Green's answer, the WebJobs SDK calls Abandon on messages that failed to process, and after maximum number of retries these messages are moved to the dead letter queue by the Service Bus. The properties defining when a message will be moved into the dead letter queue, such as maximum delivery count, time to live, and PeekLock duration can be changed in Service Bus -> Queue -> Properties.
You can find more information on SB dead letter queue here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dead-letter-queues