I'm developing two WebJobs for azure: One which will be putting messages in the Service Bus Queue using a topic and another which is subscribed to the ServiceBusTrigger using the same topic.
The messages are sent to the service bus queue correctly but when run the WebJob subscribed to the ServiceBusTrigger those messages are not being processed in FIFO basis.
The code for the WebJob which puts messages in the service bus queue is the following:
NamespaceManager namespaceManager = NamespaceManager.Create();
// Delete if exists
if (namespaceManager.TopicExists("SampleTopic"))
{
namespaceManager.DeleteTopic("SampleTopic");
}
TopicDescription td = new TopicDescription("SampleTopic");
td.SupportOrdering = true;
TopicDescription myTopic = namespaceManager.CreateTopic(td);
SubscriptionDescription myAuditSubscription = namespaceManager.CreateSubscription(myTopic.Path, "ImporterSubscription");
TopicClient topicClient = TopicClient.Create("SampleTopic");
for(int i = 1; i <= 10; i++)
{
var message = new BrokeredMessage("message"+i);
topicClient.Send(message);
}
topicClient.Close();
The WebJob which is subscrited to the service bus trigger has the following code:
namespace HO.Importer.Azure.WebJob.TGZProcessor
{
public class Program
{
static void Main(string[] args)
{
JobHostConfiguration config = new JobHostConfiguration();
config.UseServiceBus();
JobHost host = new JobHost(config);
host.RunAndBlock();
}
public static void WriteLog([ServiceBusTrigger("SampleTopic", "ImporterSubscription")] string message,
TextWriter logger)
{
Console.WriteLine(message));
}
}
}
How can I achieve to process the messages fromo the queue as FIFO?
Thanks in advance!
Use SessionId or PartitionKey, that will ensure the message is handled by the same message broker.
See: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-partitioning
"SessionId: If a message has the BrokeredMessage.SessionId property set, then Service Bus uses this property as the partition key. This way, all messages that belong to the same session are handled by the same message broker. This enables Service Bus to guarantee message ordering as well as the consistency of session states."
While Azure Service Bus provides FIFO feature(Sessions), it is better not to assume this kind of behavior with a broker based queuing system. Ben Morris had a good post Don’t assume message ordering in Azure Service Bus on the fact that assuming ordering with asynchronous messaging is almost a fallacy and reasons for that.
Related
I have a application in which I want to have 1 durable and 1 non-durable queue in Active MQ Artemis.
For connecting to this message bus I use amqpnetlite.
var source = new Source()
{
};
if (durable)
{
source.Address = amqpAddressConverter.GetSubscriberAddress(address, useLoadBalancing);
source.Durable = 1;
source.ExpiryPolicy = new Symbol("never");
source.DistributionMode = new Symbol("copy");
}
else
{
source.Address = amqpAddressConverter.GetSubscriberAddress(address);
source.Durable = 0;
source.ExpiryPolicy = "never";
}
var receiverLink = new ReceiverLink(session, linkName, source, null);
So this is my receiver link. As shown I set the Durable uint of the Source which will given into the ReceiverLink.
Because as I saw in the Active MQ Artemis documentation, that the Durable is a boolean but within the amqpnetlite library it is an uint my understanding is that everything over 0 should be true and 0 should be false.
At first the behaviour was very strange: Even when the Aretemis Web interface was shown a queue as durable it would be deleted as soon as no consumer would be connected.
I found this:
ActiveMQ Artemis queue deleted after shutdown of consuming client
which describes that even durable queues get deleted because of the default behaviour.
So I manipulated the broker.xml and set AUTO-DELETE-QUEUE to false.
Since then the behaviour completly switched:
Both (durable = 1 and durable = 0) queues are being still there after the connection disconnected.
So how to create a durable and a non-durable connection correctly?
The Artemis source carries an example in .NET that creates a durable topic subscription and also shows how to later recover it using AmqpNetLite.
One key thing many folks miss is that your client needs to use a unique container ID analogous to the JMS Client ID concept.
For Queue specific subscriptions the client should indicate in the link capabilities that it wants a Queue based address created as the default is a multicast Queue which won't behave the same.
Source source = new Source() {
Address = address,
Capabilities = new Symbol[] {"queue"},
};
vs topic specific source configuration:
Source source = new Source() {
Address = address,
Capabilities = new Symbol[] {"topic"},
};
I created a MassTransit quickstart program to interact with my localhost RabbitMQ:
namespace ConsoleApp1
{
public static class Program
{
public class YourMessage
{
public string Text { get; set; }
}
public static async Task Main(params string[] args)
{
var bus = Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://localhost"), h =>
{
h.Username("guest");
h.Password("guest");
});
sbc.ReceiveEndpoint(host, "test_queue", ep =>
{
ep.Handler<YourMessage>(async context => await Console.Out.WriteLineAsync($"Received: {context.Message.Text}"));
});
});
await bus.StartAsync();
await bus.Publish(new YourMessage{Text = "Hi"});
Console.WriteLine("Press any key to exit");
Console.ReadKey();
await bus.StopAsync();
}
}
}
Everything looked fine untill I actually checked the underlying RabbitMQ management and found out that just for this very simple program, MassTransit created 3 exchanges and 2 queues.
Exchanges, all fanouts:
ConsoleApp1:Program-YourMessage: Durable
VP0003748_dotnet_bus_6n9oyyfzxhyx9ybobdmpj8qeyt: Auto-delete and Durable?
test_queue: Durable
Queues:
VP0003748_dotnet_bus_6n9oyyfzxhyx9ybobdmpj8qeyt: x-expire 60000
test_queue: Durable
I would like to know why all of that is necessary or is the default configuration? In particular, I am not really sure to get the point of creating so "many".
It is all described in the documentation.
ConsoleApp1:Program-YourMessage is the message contract exchange, here messages are being published.
test_queue is the endpoint exchange. It binds to the message exchange. This way, when you have multiple consumers for the same message type (pub-sub), they all get their copy of the message.
test_queue is the queue, which binds to the endpoint exchange. Publish-subscribe in RMQ requires exchanges and queues can find to exchanges, so messages get properly delivered.
Both non-durable queue and exchange with weird names are the endpoint temp queue and exchange, which are used for request-response.
Just wondering the best way (even if via Portal, Powershell, or C#) to purge the messages off of a Service Bus Topic's Subscription.
Imagine we have a topic with 4 subscriptions, and we only want to purge the messages from one of the subscriptions.
I have a feeling the only way may be to read the messages in a while loop, but hoping for something better.
UPDATE:
Apart from using code, you can use the Server Explorer as suggested in the answer - right click subscription and purge messages:
You can most certainly do it via code. If you're using Service Bus SDK, you could do something like the following:
static void PurgeMessagesFromSubscription()
{
var connectionString = "Endpoint=sb://account-name.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=access key";
var topic = "topic-name";
var subscription = "subscription-name";
int batchSize = 100;
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, topic, subscription, ReceiveMode.ReceiveAndDelete);
do
{
var messages = subscriptionClient.ReceiveBatch(batchSize);
if (messages.Count() == 0)
{
break;
}
}
while (true);
}
What this code will do is fetch messages from the subscription (100 at a time) in Receive & Delete mode so that as soon as messages are fetched, they are deleted from the subscription automatically.
I believe Service Bus Explorer tool also has the capability to purge messages. You can use that as well instead of writing the code.
If you have a lot of messages and can tolerate a bit of downtime on subscriber side, it might be faster to just drop the subscription and create a new one with the same name.
Thank you #Gaurav Mantri, I used slightly changed code without the batch option with version 5.2.0 of Microsoft.Azure.ServiceBus Nuget Package:
var connectionString = "Endpoint=sb://";
var topic = "topic";
var subscription = "subscription";
var subscriptionClient = new SubscriptionClient(connectionString, topic, subscription, ReceiveMode.ReceiveAndDelete);
subscriptionClient.RegisterMessageHandler(
(message, token) =>
{
Console.WriteLine($"Received message: SequenceNumber:
{message.SystemProperties.SequenceNumber}");
return Task.CompletedTask;
},
(exceptionEvent) =>
{
Console.WriteLine("Exception = " + exceptionEvent.Exception);
return Task.CompletedTask;
});
I have created a azure webjob which will send a strongly typed message to a service bus queue and it successfully sends.
I want to create another webjob which should be triggered whenever there is a message in the servicebus queue. Please find below the code i am trying. For some reason, though there are messages in the servicebus queue, the webjob is not getting triggered and getting an error when i run the webjob locally.
Error:
System.InvalidOperationException
{"Missing value for trigger parameter 'blobIinfo'."}
Code:
public static void Main()
{
var config = new JobHostConfiguration
{
NameResolver = new QueueNameResolver(),
ServiceBusConnectionString = ApplicationSettings.ServiceBusConnectionString
};
var host = new JobHost(config);
host.Call(typeof(BankLineFileProcessorWebJob).GetMethod("ProcessQueueMessage"));
}
[NoAutomaticTrigger]
public static void ProcessQueueMessage(
TextWriter log,
[ServiceBusTrigger("testsftppollingqueue")] SftpQueueMessage blobIinfo
)
{
while (true)
{
log.WriteLine("Queue message refers to blob: " + blobIinfo.BlobUri);
Thread.Sleep(TimeSpan.FromMinutes(PollingInterval));
}
}
Can anyone help me how to solve this?
Thanks
You have to use
host.RunAndBlock();
instead of
host.Call(typeof(BankLineFileProcessorWebJob).GetMethod("ProcessQueueMessage"));
Also, please take out the NoAutomaticTrigger attribute.
Since we are experiencing some trouble with IBM's Websphere MQ using XMS.net (Windows service that sometimes seems to give up listening for messages on a queue) we would like to create a simple application to monitor the depths of some queues (or number of messages on the queue) to be able to alert someone when the queue depth exceeds a certain threshold. This application would be launched by the task scheduler on a specific interval and would "read out" for X queues their queue depth (and maybe some other statistics).
Our windows service is using the following code and I was hoping I could reuse that same "knowledge" for our "monitoring" application.
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
//Read config values
string QueueManager = ConfigurationManager.AppSettings["queuemanager"];
string Channel = ConfigurationManager.AppSettings["channel"];
string Queue = ConfigurationManager.AppSettings["queue"];
string HostIP = ConfigurationManager.AppSettings["host"];
int Port = int.Parse(ConfigurationManager.AppSettings["port"]);
//Create connection
var factoryfactory = XMSFactoryFactory.GetInstance(XMSC.CT_WMQ);
var connectionfactory = factoryfactory.CreateConnectionFactory();
connectionfactory.SetStringProperty(XMSC.WMQ_QUEUE_MANAGER, QueueManager);
connectionfactory.SetStringProperty(XMSC.WMQ_HOST_NAME, HostIP);
connectionfactory.SetIntProperty(XMSC.WMQ_PORT, Port);
connectionfactory.SetStringProperty(XMSC.WMQ_CHANNEL, Channel);
connectionfactory.SetIntProperty(XMSC.WMQ_BROKER_VERSION, XMSC.WMQ_BROKER_V2);
connectionfactory.SetIntProperty(XMSC.WMQ_CONNECTION_MODE, XMSC.WMQ_CM_CLIENT_UNMANAGED);
Console.WriteLine("Creating connection");
var connection = connectionfactory.CreateConnection();
connection.ExceptionListener = new ExceptionListener(OnXMSExceptionReceived);
//Create a_session
Console.WriteLine("Creating sessions");
var session = connection.CreateSession(false, AcknowledgeMode.ClientAcknowledge);
//Create queue
Console.WriteLine("Creating queue");
var queue = session.CreateQueue(string.Format("queue://{0}/{1}", QueueManager, Queue));
I have browsed through the properties of session, queue etc. but, ofcourse, there are no "current queue depth" properties. I could use GetIntProperty() or GetLongProperty() on these objects but I don't know which constant to use for that (I have seen IBM.XMS.MQC.MQIA_CURRENT_Q_DEPTH but that contains an int and Get...Property() expects a string as parameter).
Long story short: how would I go about retrieving a queues depth with the above code as a starting-point? Is it at all possible using XMS.Net?
I was able to solve it using, as Shashi suggested, the MQ API. For this you need to reference amqmdnet.dll (C:\Program Files (x86)\IBM\WebSphere MQ\bin\amqmdnet.dll) and use the following (example) code. Please note that this is a simple example, no exception handling etc. is included.
using System;
using System.Collections;
using System.Configuration;
using IBM.WMQ;
namespace Test
{
class Program
{
static void Main(string[] args)
{
//Connection properties
var properties = new Hashtable();
properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
properties.Add(MQC.CHANNEL_PROPERTY, "SOME.CHANNEL.TCP");
properties.Add(MQC.HOST_NAME_PROPERTY, "12.34.56.78");
properties.Add(MQC.PORT_PROPERTY, 1416);
var qmgr = new MQQueueManager("MYQMGR", properties);
Console.WriteLine("Local : {0}", GetQueueDepth(qmgr, "FOO.LOCALQ"));
Console.WriteLine("Report : {0}", GetQueueDepth(qmgr, "FOO.REPORTQ"));
}
public static int GetQueueDepth(MQQueueManager queuemgr, string queue)
{
return queuemgr.AccessQueue(queue,
MQC.MQOO_INPUT_AS_Q_DEF +
MQC.MQOO_FAIL_IF_QUIESCING +
MQC.MQOO_INQUIRE).CurrentDepth;
}
}
}
This performs way better than my initial "workaround".
Using XMS .NET queue depth can't be determined. Queue depth is specific to messaging providers and not JMS/XMS, so you will need to use MQ APIs to get the queue depth. You could use MQ .NET API to find the queue depth. MQQueue.CurrentDepth will give the number of message in the queue.
IMO it would be good to investigate why XMS .NET service stopped listening for messages than write another program to monitor queue depth.