Develop Reference

The correct way to wait for a specific message in a Service Bus Queue in a multi threaded environment (Azure Functions)

I have created a solution based on Azure Functions and Azure Service Bus, where clients can retrieve information from multiple back-end systems using a single API. The API is implemented in Azure Functions, and based on the payload of the request it is relayed to a Service Bus Queue, picked up by a client application running somewhere on-premise, and the answer sent back by the client to another Service Bus Queue, the "reply-" queue. Meanwhile, the Azure Function is waiting for a message in the reply-queue, and when it finds the message that belongs to it, it sends the payload back to the caller.
The Azure Function Activity Root Id is attached to the Service Bus Message as the CorrelationId. This way each running function knows which message contains the response to the callers request.
My question is about the way I am currently retrieving the messages from the reply queue. Since multiple instances can be running at the same time, each Azure Function instance needs to get it's response from the client without blocking other instances. Besides that, a time out needs to be observed. The client is expected to respond within 20 seconds. While waiting, the Azure Function should not be blocking other instances.
This is the code I have so far:
internal static async Task<(string, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var receiver = new MessageReceiver(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var sw = Stopwatch.StartNew();
while (sw.Elapsed < timeout)
{
var message = await receiver.ReceiveAsync(timeout.Subtract(sw.Elapsed));
if (message != null)
{
if (message.CorrelationId == operationId)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
else
{
log.LogInformation("Ignoring message for operation {OperationId}", message.CorrelationId);
}
}
}
return (null, false);
}
finally
{
await receiver.CloseAsync();
}
}
The code is based on a few assumptions. I am having a hard time trying to find any documentation to verify my assumptions are correct:
I expect subsequent calls to ReceiveAsync not to fetch messages I have previously fetched and not explicitly abandoned.
I expect new messages that arrive on the queue to be received by ReceiveAsync, even though they may have arrived after my first call to ReceiveAsync and even though there might still be other messages in the queue that I haven't received yet either. E.g. there are 10 messages in the queue, I start receiving the first few message, meanwhile new messages arrive, and after I have read the 10 pre-existing messages, I get the new messages too.
I expect that when I call ReceiveAsync for a second time, that the lock is released from the message I received with the first call, although I did not explicitly Abandon that first message.
Could anyone tell me if my assumptions are correct?
Note: please don't suggest that Durable Functions where designed specifically for this, because they simply do not fill the requirements. Most notably, Durable Functions are invoked by a process that polls a queue with a sliding interval, so after not having any requests for a few minutes, the first new request can take a minute to start, which is not acceptable for my use case.

I would consider session enabled topics or queues for this.
The Message sessions documentation explains this in detail but the essential bit is that a session receiver is created by a client accepting a session. When the session is accepted and held by a client, the client holds an exclusive lock on all messages with that session's session ID in the queue or subscription. It will also hold exclusive locks on all messages with the session ID that will arrive later.
This makes it perfect for facilitating the request/reply pattern.
When sending the message to the queue that the on-premises handlers receive messages on, set the ReplyToSessionId property on the message to your operationId.
Then, the on-premises handlers need to set the SessionId property of the messages they send to the reply queue to the value of the ReplyToSessionId property of the message they processed.
Then finally you can update your code to use a SessionClient and then use the 'AcceptMessageSessionAsync()' method on that to start listening for messages on that session.
Something like the following should work:
internal static async Task<(string?, bool)> WaitForMessageAsync(string queueName, string operationId, TimeSpan timeout, ILogger log)
{
log.LogInformation("Connecting to service bus queue {QueueName} to wait for reply...", queueName);
var sessionClient = new SessionClient(_connectionString, queueName, ReceiveMode.PeekLock);
try
{
var receiver = await sessionClient.AcceptMessageSessionAsync(operationId);
// message will be null if the timeout is reached
var message = await receiver.ReceiveAsync(timeout);
if (message != null)
{
log.LogInformation("Reply received for operation {OperationId}", message.CorrelationId);
var reply = Encoding.UTF8.GetString(message.Body);
var error = message.UserProperties.ContainsKey("ErrorCode");
await receiver.CompleteAsync(message.SystemProperties.LockToken);
return (reply, error);
}
return (null, false);
}
finally
{
await sessionClient.CloseAsync();
}
}
Note: For all this to work, the reply queue will need Sessions enabled. This will require the Standard or Premium tier of Azure Service Bus.
Both queues and topic subscriptions support enabling sessions. The topic subscriptions allow you to mix and match session enabled scenarios as your needs arise. You could have some subscriptions with it enabled, and some without.
The queue used to send the message to the on-premises handlers does not need Sessions enabled.
Finally, when Sessions are enabled on a queue or a topic subscription, the client applications can no longer send or receive regular messages. All messages must be sent as part of a session (by setting the SessionId) and received by accepting the session.

It seems that the feature can not be achieved now.
You can give your voice here where if others have same demand, they will vote up your idea.

Does ReceiveAsync Method in 'Microsoft.Azure.Devices.Client' Consumes Internet?

I am using Azure SDKs on IoT devices. One of the methods I rely on is
public Task<Message> ReceiveAsync();
which appears in this namespace
namespace Microsoft.Azure.Devices.Client
Under this class
public sealed class DeviceClient : IDisposable
I am calling this method continuously within a while loop as follows
while (true)
{
var receivedMessage = await _deviceClient.ReceiveAsync(TimeSpan.FromSeconds(3)).ConfigureAwait(false);
if (receivedMessage != null)
{
//Do staff
}
}
My question is: does this consume internet quotas even though the receivedMessage always shows null?

Digging through the source, you'll find three handlers:
HttpTransportHandler
MqttTransportHandler
AmqpTransportHandler
Which one is used, depends on your configuration. The HTTP one will issue a GET request per ReceiveAsync(), costing network traffic.
The MQTT handler operates on TCP or WebSockets, where keepalive traffic may be involved. But given this communication is bidirectional, most traffic that occurs involves actual messages being delivered. ReceiveAsync() simply gets the first message from the internal receive queue, if any, or waits for one to arrive, it doesn't poll.
The AMPQ handler also operates on a message queue, and I can't quite figure out whether a ReceiveAsync() will ultimately incur network traffic.

UDP Multicast Sending (using JoinMulticastGroup) in C#

I know there are plenty of examples around the web regarding UDP multicasting in C#. This is more to get a clarification on the need to include the method JoinMulticastGroup when sending only. Most code examples I have come across nearly always include this method as part of the initialisation code. But surely if the program or class is only ever sending, then it is not required?
i.e. on another stackoverflow question someone uses the code
public void SendMessage(string message)
{
var data = Encoding.Default.GetBytes(message);
using (var udpClient = new UdpClient(AddressFamily.InterNetwork))
{
var address = IPAddress.Parse("224.100.0.1");
var ipEndPoint = new IPEndPoint(address, 8088);
udpClient.JoinMulticastGroup(address);
udpClient.Send(data, data.Length, ipEndPoint);
udpClient.Close();
}
}
Is the line udpClient.JoinMulticastGroup(address); not actually redundant in this case?

JoinMulticastGroup is indeed for enabling the socket to receive multicast packets destined for that group address. If your client is only sending, then it's not strictly necessary.
However, it doesn't hurt, and does help make the code clear that you're "part of" that multicast group. In this way, if the requirements change in the future, and this application needs to receive packets, then it will already be part of the multicast group.
A source host sends data to a multicast group by simply setting the destination IP address of the datagram to be the multicast group address. Any host can become a source and send data to a multicast group. Sources do not need to register in any way before they can begin sending data to a group, and do not need to be members of the group themselves.
-- metaswitch.com

Cant consume message from Topic in activemq

I am new to activemq. T want to ask a question about the topics of Activemq. I succeed to get a message from a queue. Also I can send message to topic/Queue, but I can't get a message from Topic.
I have tried using Java Code. The result is the same.
The following is my core code:
connection.ClientId = clientId;
connection.Start();
using (ISession session = connection.CreateSession())
{
ITopic topic = new Apache.NMS.Commands.Topic(topicName);
IDestination destination = SessionUtil.GetDestination(session, topicName,
DestinationType.Topic);
using (IMessageConsumer consumer = **session.CreateDurableConsumer**(topic, "news", null, false))
{
**consumer.Listener += new MessageListener(consumer_Listener);**
//**IMessage iMsg = consumer.Receive();**
// if (iMsg != null)//{
// ITextMessage msg = (ITextMessage)iMsg;
// return msg.Text;
// }
//else
//return iMsg;
}
}
I also using: IMessage iMsg = consumer.Receive();
IMsg　always null（topicname has messages. How can I consume topic's message?

The Messages would need to have been sent after the Topic consumer was created. A Topic is fire and forget, if there are no consumers then the message is discarded. Any consumer that comes online will only receive message sent after that time unless it is either a Durable Topic consumer or a Queue consumer.
In the case of a durable consumer you must have created an instance of it so there is a subscription record before those message were sent to the Topic. So I would guess your problem is that you didn't subscribe this consumer before and so the Broker was not storing any Messages for it.

I was so stupid about the phrase "using".Beacause I use "using" open connection and session. when the code block was excuted, the connnection/session is disappear. Now I dont use "using" block to cerate connection. just like normal code. It works. also I build "Global.asax" file. The program can listener Topic once started up. At the same time, I write a function to colse the connection.I tested. Once a message was sent to the topic, the Onessage() function would be exectued.
just resolve my problem.maybe you would have better answer.Thanks Tim.

ActiveMQ 5.7.0 Selector not working in C#

I have a very simple ActiveMQ message consumer that's created in C# as follows:
using(IMessageConsumer consumer = session.CreateConsumer(destination,"NMSCorrelationID='<value of correlation id>'")){
/* This Receive(..) operation does not retrieve the message with the correlation id which I confirmed to be available on the queue. */
IMessage message = consumer.Receive(new TimeSpan(1000));
}
However, I can get the message if I don't use the selector while creating the consumer. The destination is a queue on the ActiveMQ broker. I've tried using CorrelationID and JMSCorrelationID as the selectors, but none on them worked. The ActiveMQ broker was installed with out-of-the-box settings. Is there any special setting that I need to use for the selectors to work?

You definitely want to set the selector with JMSCorrelationID. Using NMSCorrelationID, or just CorrelationID will cause it to ignore all of the messages. I tested the following out with both topics and queues, and everything worked correctly. I did test on ActiveMQ 5.8.0, but I'm pretty sure this will work just fine on 5.7.0.
IMessageConsumer subscriber = session.CreateConsumer(
"queue://TestCorrelation",
"JMSCorrelationID = 'FOO'",
false);
The broker will not enqueue a message to a consumer from the same connection as the producer if that consumer has set the third parameter (noLocal) to true. You will need to have two separate connections to get the correlation ID selector to work. One to send the message, a consumer on another connection to receive the message. If you set noLocal to false, then a consumer on the same connection as the producer will receive the message.
You can also try using some wildcards in the selector if you want to test.
"JMSCorrelationID LIKE '%FOO%'"
Be aware that the selector is case sensitive. Your correlation IDs must match exactly.