Develop Reference

The correct way to wait for a specific message in a Service Bus Queue in a multi threaded environment (Azure Functions)

I have created a solution based on Azure Functions and Azure Service Bus, where clients can retrieve information from multiple back-end systems using a single API. The API is implemented in Azure Functions, and based on the payload of the request it is relayed to a Service Bus Queue, picked up by a client application running somewhere on-premise, and the answer sent back by the client to another Service Bus Queue, the "reply-" queue. Meanwhile, the Azure Function is waiting for a message in the reply-queue, and when it finds the message that belongs to it, it sends the payload back to the caller.
The Azure Function Activity Root Id is attached to the Service Bus Message as the CorrelationId. This way each running function knows which message contains the response to the callers request.
My question is about the way I am currently retrieving the messages from the reply queue. Since multiple instances can be running at the same time, each Azure Function instance needs to get it's response from the client without blocking other instances. Besides that, a time out needs to be observed. The client is expected to respond within 20 seconds. While waiting, the Azure Function should not be blocking other instances.
This is the code I have so far:
internal static async Task<(string, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var receiver = new MessageReceiver(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var sw = Stopwatch.StartNew();
while (sw.Elapsed < timeout)
{
var message = await receiver.ReceiveAsync(timeout.Subtract(sw.Elapsed));
if (message != null)
{
if (message.CorrelationId == operationId)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
else
{
log.LogInformation("Ignoring message for operation {OperationId}", message.CorrelationId);
}
}
}
return (null, false);
}
finally
{
await receiver.CloseAsync();
}
}
The code is based on a few assumptions. I am having a hard time trying to find any documentation to verify my assumptions are correct:
I expect subsequent calls to ReceiveAsync not to fetch messages I have previously fetched and not explicitly abandoned.
I expect new messages that arrive on the queue to be received by ReceiveAsync, even though they may have arrived after my first call to ReceiveAsync and even though there might still be other messages in the queue that I haven't received yet either. E.g. there are 10 messages in the queue, I start receiving the first few message, meanwhile new messages arrive, and after I have read the 10 pre-existing messages, I get the new messages too.
I expect that when I call ReceiveAsync for a second time, that the lock is released from the message I received with the first call, although I did not explicitly Abandon that first message.
Could anyone tell me if my assumptions are correct?
Note: please don't suggest that Durable Functions where designed specifically for this, because they simply do not fill the requirements. Most notably, Durable Functions are invoked by a process that polls a queue with a sliding interval, so after not having any requests for a few minutes, the first new request can take a minute to start, which is not acceptable for my use case.

I would consider session enabled topics or queues for this.
The Message sessions documentation explains this in detail but the essential bit is that a session receiver is created by a client accepting a session. When the session is accepted and held by a client, the client holds an exclusive lock on all messages with that session's session ID in the queue or subscription. It will also hold exclusive locks on all messages with the session ID that will arrive later.
This makes it perfect for facilitating the request/reply pattern.
When sending the message to the queue that the on-premises handlers receive messages on, set the ReplyToSessionId property on the message to your operationId.
Then, the on-premises handlers need to set the SessionId property of the messages they send to the reply queue to the value of the ReplyToSessionId property of the message they processed.
Then finally you can update your code to use a SessionClient and then use the 'AcceptMessageSessionAsync()' method on that to start listening for messages on that session.
Something like the following should work:
internal static async Task<(string?, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var sessionClient = new SessionClient(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var receiver = await sessionClient.AcceptMessageSessionAsync(operationId);
// message will be null if the timeout is reached
var message = await receiver.ReceiveAsync(timeout);
if (message != null)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
return (null, false);
}
finally
{
await sessionClient.CloseAsync();
}
}
Note: For all this to work, the reply queue will need Sessions enabled. This will require the Standard or Premium tier of Azure Service Bus.
Both queues and topic subscriptions support enabling sessions. The topic subscriptions allow you to mix and match session enabled scenarios as your needs arise. You could have some subscriptions with it enabled, and some without.
The queue used to send the message to the on-premises handlers does not need Sessions enabled.
Finally, when Sessions are enabled on a queue or a topic subscription, the client applications can no longer send or receive regular messages. All messages must be sent as part of a session (by setting the SessionId) and received by accepting the session.

It seems that the feature can not be achieved now.
You can give your voice here where if others have same demand, they will vote up your idea.

RabbitMQ Verify Message Was Sent

This question is already exists and answered. But there is a dark side in answers. My channel already supports BasicAcks and BasicNacks handlers (in a poor way):
Channel.BasicAcks += (sender, eventArgs) =>
{
Console.WriteLine("Basic Ack!");
}
Channel.BasicNacks += (sender, eventArgs) =>
{
Console.WriteLine("Basic Nack!");
}
I have a message that published to a queue. so I use this code to do that:
Channel.BasicPublish("ExchangeName", "QueueName", messageProperties, payload);
Channel.WaitForConfirmOrDie();
As long as WaitForConfirmOrDie is a void function, how can I know if message received by a queue? Or more precise, how can I implement Ack handlers to give me a clear state of published message in order to not send it again to queue or in the case of BasicNack send it again?

Using the BasicAcks and BasicNacks event handlers is independent of calling Channel.WaitForConfirmOrDie.
Channel.WaitForConfirmOrDie is a convenience method that synchronously waits for message acknowledgements. So, if you publish messages one-by-one, you will wait for these acks one-by-one. As you can imagine, that is pretty inefficient.
What you should do is register for BasicAcks and BasicNacks like you have done. You should have an "acceptable number of outstanding confirms" defined. Here's one way to implement this -
Publish up to N messages without an ack/nack (N is up to you). If the next message would exceed N do not continue to publish messages.
While a message is outstanding, save it locally (in RAM or local disk). Remember that you can't be 100% sure a message is queued until you get an ack for it.
If the message is acked, remove it from local storage and decrease the count of outstanding messages, which allows publishing to continue (if publishing is blocked). Please remember that messages can be acked in batches.
If the message is nacked, you could re-try it up to a certain number of times, maybe with backoff. Once the re-try limit is exceeded, raise an application exception.

How to get size (number of messages) of a MassTransit IBus?

I'm using the MassTransit library's InMemoryMessageBus and I would like to know how I can get the number of messages in the queue (the size of the bus).

The number of messages in any particular queue using the in-memory transport is not available. The message delivery is based on a queued task scheduler, and the message counts have not been made available. I'm not sure if they could be or not (well, easily. Anything is possible, but practical is another matter).
UPDATE: This was added to MassTransit and will be in the next release (3.5.x). The tracking issue is on GitHub, including example usage of the new code.

If you are using RabbitMQ as your transport, You could use HareDu.
The following snippet will get you started:
var client = HareDuFactory.New(x => x.ConnectTo(RabbitMqHostUrl));
var data = client
.Factory<VirtualHostResources>(y => y.Credentials(RabbitMqUser, RabbitMqPass))
.Queue
.GetAll()
.Data();
foreach (var queue in data)
{
/*then you can access
queue.Name, queue.VirtualHostName, queue.Memory, queue.Messages,
queue.MessagesReady, queue.MessagesUnacknowledged, queue.Node, queue.IsDurable, queue.Consumers, queue.IdleSince */
}

Odd Behavior of Azure Service Bus ReceiveBatch()

Working with a Azure Service Bus Topic currently and running into an issue receiving my messages using ReceiveBatch method. The issue is that the expected results are not actually the results that I am getting. Here is the basic code setup, use cases are below:
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(connectionString, convoTopic, subName);
IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100);
foreach (BrokeredMessage message in messageList)
{
try
{
Console.WriteLine(message.GetBody<string>() + message.MessageId);
message.Complete();
}
catch (Exception ex)
{
message.Abandon();
}
}
client.Close();
MessageBox.Show("Done");
Using the above code, if I send 4 messages, then poll on the first run through I get the first message. On the second run through I get the other 3. I'm expecting to get all 4 at the same time. It seems to always return a singular value on the first poll then the rest on subsequent polls. (same result with 3 and 5 where I get n-1 of n messages sent on the second try and 1 message on the first try).
If I have 0 messages to receive, the operation takes between ~30-60 seconds to get the messageList (that has a 0 count). I need this to return instantly.
If I change the code to IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100, new Timespan(0,0,0)); then issue #2 goes away because issue 1 still persists where I have to call the code twice to get all the messages.
I'm assuming that issue #2 is because of a default timeout value which I overwrite in #3 (though I find it confusing that if a message is there it immediately responds without waiting the default time). I am not sure why I never receive the full amount of messages in a single ReceiveBatch however.

The way I got ReceiveBatch() to work properly was to do two things.
Disable Partitioning in the Topic (I had to make a new topic for this because you can't toggle that after creation)
Enable Batching on each subscription created like so:
List item
SubscriptionDescription sd = new SubscriptionDescription(topicName, orgSubName);
sd.EnableBatchedOperations = true;
After I did those two things, I was able to get the topics to work as intended using IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100, new TimeSpan(0,0,0));

I'm having a similar problem with an ASB Queue. I discovered that I could mitigate it somewhat by increasing the PrefetchCount on the client prior to receiving the batch:
SubscriptionClient client = SubscriptionClient.CreateFromConnectionString(connectionString, convoTopic, subName);
client.PrefetchCount = 100;
IEnumerable<BrokeredMessage> messageList = client.ReceiveBatch(100);
From the Azure Service Bus Best Practices for Performance Improvements Using Service Bus Brokered Messaging:
Prefetching enables the queue or subscription client to load additional messages from the service when it performs a receive operation.
...
When using the default lock expiration of 60 seconds, a good value for
SubscriptionClient.PrefetchCount is 20 times the maximum processing rates of all receivers of the factory. For example, a factory creates 3 receivers, and each receiver can process up to 10 messages per second. The prefetch count should not exceed 20*3*10 = 600.
...
Prefetching messages increases the overall throughput for a queue or subscription because it reduces the overall number of message operations, or round trips. Fetching the first message, however, will take longer (due to the increased message size). Receiving prefetched messages will be faster because these messages have already been downloaded by the client.

Just a few more pieces to the puzzle. I still couldn't get it to work even after Enable Batching and Disable Partitioning - I still had to do two ReceiveBatch calls. I did find however:
Restarting the Service Bus services (I am using Service Bus for Windows Server) cleared up the issue for me.
Doing a single RecieveBatch and taking no action (letting the message locks expire) and then doing another ReceiveBatch caused all of the messages to come through at the same time. (Doing an initial ReceiveBatch and calling Abandon on all of the messages didn't cause that behavior.)
So it appears to be some sort of corruption/bug in Service Bus's in-memory cache.

brokeredmessage microsoft service bus queue ReceiveBatch not obtaining all dead letter messages

I am testing a project with a dead letter queue with Microsoft Service Bus. I send 26 messages (representing the alphabet) and I use a program that when receiving the messages, randomly puts some of them in a dead letter queue. The messages are always read in peek mode from the dead letter queue, so once they reach there they stay there. After running a few times, all 26 messages will be in the dead letter queue, and always remain there.
However, when reading them, sometimes only a few (e.g. 6) are read, sometimes all 26.
I use the command:
const int maxToRead = 200; // It seems one wants to set this higher than
// the anticipated load, obtaining only some back
IEnumerable<BrokeredMessage> dlIE =
deadletterSubscriptionClient.ReceiveBatch(maxToRead);
There is an overload of ReceiveBatch which has a timeout, but this doesn't help, and proably only adds to the complexity.
Why doesn't it obtain all 26 messages every time, since it is used in "peek" mode and the messages stay there.
I can use "Service Bus Explorer" to actually verify that all messages are in the deadletter queue and remain there.
This is mostly a testing example, but one would hope that "ReceiveBatch" would work in deterministic mode and not in a very (bad) random manner...

This is only a partial-answer or work-around; the following code reliably gets all elements, but doesn't use the "ReceiveBatch"; note, as far as I can discern, Peek(i) operates on a one-based index. Also: depending on which server one is running on, if you are charged by the message pull, this may (or may not) be more expensive, so use at your own risk:
List<BrokeredMessage> dlIE = new List<BrokeredMessage>();
BrokeredMessage potentialMessage = null;
int loopCount = 1;
while ((potentialMessage = deadletterSubscriptionClient.Peek(loopCount)) != null)
{
dlIE.Add(potentialMessage);
loopCount++;
}