Develop Reference

RabbitMQ consuming one message before receiving a new

If I sends the message to a task inside my service and returns from the RabbitMQ “Received” callback I get a new callback immediately. This is not intended since my service then will work as a new queue and makes spawning of a new worker more less useless since the first service has dequeued all messages.
I can see in the python examples, that the callback acks the message when it’s done but the C# doesn’t. Can this ack bed used for blocking for new messages until the current message is processed.
Also finds blocking the “Received” event function with a semaphore until processing is done very very hacky.

You need to set consumer prefetch to 1: https://www.rabbitmq.com/consumer-prefetch.html
Ack the message when you're done with it, and you'll get the next one.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

Azure function: limit the number of calls per second

I have an Azure function triggered by queue messages. This function makes a request to third-party API. Unfortunately this API has limit - 10 transactions per second, but I might have more than 10 messages per second in service bus queue. How can I limit the number of calls of Azure function to satisfy third-party API limitations?

Unfortunately there is no built-in option for this.
The only reliable way to limit concurrent executions would be to run on a fixed App Service Plan (not Consumption Plan) with just 1 instance running all the time. You will have to pay for this instance.
Then set the option in host.json file:
"serviceBus": {
// The maximum number of concurrent calls to the callback the message
// pump should initiate. The default is 16.
"maxConcurrentCalls": 10
}
Finally, make sure your function takes a second to execute (or other minimal duration, and adjust concurrent calls accordingly).
As #SeanFeldman suggested, see some other ideas in this answer. It's about Storage Queues, but applies to Service Bus too.

You can try writing some custom logic i.e. implement your own in-memory queue in Azure function to queue up requests and limit the calls to third party API. Anyway until the call to third party API succeeds, you dont need to acknowledge the messages in the queue. In this way reliability is also maintained.

The best way to maintain integrity of the system is to throttle the consumption of the Service Bus messages. You can control how your QueueClient processes the messages, see: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues#4-receive-messages-from-the-queue
Check out the "Max Concurrent calls"
static void RegisterOnMessageHandlerAndReceiveMessages()
{
// Configure the message handler options in terms of exception handling, number of concurrent messages to deliver, etc.
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
// Maximum number of concurrent calls to the callback ProcessMessagesAsync(), set to 1 for simplicity.
// Set it according to how many messages the application wants to process in parallel.
MaxConcurrentCalls = 1,
// Indicates whether the message pump should automatically complete the messages after returning from user callback.
// False below indicates the complete operation is handled by the user callback as in ProcessMessagesAsync().
AutoComplete = false
};
// Register the function that processes messages.
queueClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}

Do you want to get rid of N-10 messages you receive in a second interval or do you want to treat every message in respect to the API throttling limit? For the latter, you can add the messages processed by your function to another queue from which you can read a batch of 10 messages via another function (timer trigger) every second

Apache NMS Getting pending message count

I am trying to get the current number of messages on an activeMQ queue using c#.
I have found this link (that is quite old now)
ActiveMQ with C# and Apache NMS - Count messages in queue
but enumerating the queue seems like a lot of work for this simple task.
Is this the only way to get the queue message count? If I do use this method is the queue locked while I am enumerating (I don't want to block other readers)?
Thanks,
Nick

You can either do the enumeration thing described in that other answer which won't get you the correct answer in many cases or you can use the statistics broker plugin and query that data from the broker.
With the statistics plugin you can send a message to a control queue and listen for a response on the replyTo destination you provide and get the full statistics of the destination, the caveat being that you need to parse out the data but that shouldn't be that hard.
The enumeration method won't lock the queue but it won't work the way you want because there is a limit to the depth that the broker will go into a deep queue before stopping when it is feeding a QueueBrowser so you can't be sure you got a correct count. Also using the statistic plugin results in less broker overhead and network traffic since the broker only has to send you one response with the data in it verses sending you all the messages just for the sake of counting.

Azure Service Bus Subscriber regularly phoning home?

We have pub/sub application that involves an external client subscribing to a Web Role publisher via an Azure Service Bus Topic. Our current billing cycle indicates we've sent/received >25K messages, while our dashboard indicates we've sent <100. We're investigating our implementation and checking our assumptions in order to understand the disparity.
As part of our investigation we've gathered wireshark captures of client<=>service bus traffic on the client machine. We've noticed a regular pattern of communication that we haven't seen documented and would like to better understand. The following exchange occurs once every 50s when there is otherwise no activity on the bus:
The client pushes ~200B to the service bus.
10s later, the service bus pushes ~800B to the client. The client registers the receipt of an empty message (determined via breakpoint.)
The client immediately responds by pushing ~1000B to the service bus.
Some relevant information:
This occurs when our web role is not actively pushing data to the service bus.
Upon receiving a legit message from the Web Role, the pattern described above will not occur again until a full 50s has passed.
Both client and server connect to sb://namespace.servicebus.windows.net via TCP.
Our application messages are <64 KB
Questions
What is responsible for the regular, 3-packet message exchange we're seeing? Is it some sort of keep-alive?
Do each of the 3 packets count as a separately billable message?
Is this behavior configurable or otherwise documented?
EDIT:
This is the code the receives the messages:
private void Listen()
{
_subscriptionClient.ReceiveAsync().ContinueWith(MessageReceived);
}
private void MessageReceived(Task<BrokeredMessage> task)
{
if (task.Status != TaskStatus.Faulted && task.Result != null)
{
task.Result.CompleteAsync();
// Do some things...
}
Listen();
}

I think what you are seeing is the Receive call in the background. Behind the scenes the Receive calls are all using long polling. Which means they call out to the Service Bus endpoint and ask for a message. The Service Bus service gets that request and if it has a message it will return it immediately. If it doesn't have a message it will hold the connection open for a time period in case a message arrives. If a message arrives within that time frame it will be returned to the client. If a message is not available by the end of the time frame a response is sent to the client indicating that no message was there (aka, your null BrokeredMessage). If you call Receive with no overloads (like you've done here) it will immediately make another request. This loop continues to happend until a message is received.
Thus, what you are seeing are the number of times the client requests a message but there isn't one there. The long polling makes it nicer than what the Windows Azure Storage Queues have because they will just immediately return a null result if there is no message. For both technologies it is common to implement an exponential back off for requests. There are lots of examples out there of how to do this. This cuts back on how often you need to go check the queue and can reduce your transaction count.
To answer your questions:
Yes, this is normal expected behaviour.
No, this is only one transaction. For Service Bus you get charged a transaction each time you put a message on a queue and each time a message is requested (which can be a little opaque given that Recieve makes calls multiple times in the background). Note that the docs point out that you get charged for each idle transaction (meaning a null result from a Receive call).
Again, you can implement a back off methodology so that you aren't hitting the queue so often. Another suggestion I've recently heard was if you have a queue that isn't seeing a lot of traffic you could also check the queue depth to see if it was > 0 before entering the loop for processing and if you get no messages back from a receive call you could go back to watching the queue depth. I've not tried that and it is possible that you could get throttled if you did the queue depth check too often I'd think.
If these are your production numbers then your subscription isn't really processing a lot of messages. It would likely be a really good idea to have a back off policy to a time that is acceptable to wait before it is processed. Like, if it is okay that a message sits for more than 10 minutes then create a back off approach that will eventually just be checking for a message every 10 minutes, then when it gets one process it and immediately check again.
Oh, there is a Receive overload that takes a timeout, but I'm not 100% that is a server timeout or a local timeout. If it is local then it could still be making the calls every X seconds to the service. I think this is based on the OperationTimeout value set on the Messaging Factory Settings when creating the SubscriptionClient. You'd have to test that.

How can I throttle the amount of messages coming from ActiveMQ in my C# app?

I'm using ActiveMQ in a .Net program and I'm flooded with message-events.
In short when I get a queue-event 'onMessage(IMessage receivedMsg)' I put the message into an internal queue out of which X threads do their thing.
At first I had: 'AcknowledgementMode.AutoAcknowledge' when creating the session so I'm guessing that all the messages in the queue got sucked down and put into the memory queue (which is risky since with a crash, everything is lost).
So then I used: 'AcknowledgementMode.ClientAcknowledge' when creating the session, and when a worker was ready with the message it calls the 'commit()' method on the message. However, still all the messages get sucked down from the queue.
How can I configure it that ONLY an X amount of messages are being processed or are in an internal queue, and that not everything is being 'downloaded' right away?

Are you on .NET 4.0? You could use a BlockingCollection . Set it to the maximum amount it may contain. As soon as a thread tries to put in an excess element, the Add operation will block until the collection falls below the threshold again.
Maybe that would do it for throttling?
There is also an API for throttling in the Rx framework, but I do not know how it is implemented. If you implement your Queue source as Observable, this API would become available for you, but I don't know if this hits your needs.

You can set the client prefetch to control how many messages the client will be sent. When the Session is in Auto Ack, the client will only ack a message once its been delivered to your app via the onMessage callback or through a synchronous receive. By default the client will prefetch 1000 messages from the broker, if the client goes down these messages would be redelivered to another client it this was a Queue, otherwise for a topic they are just discarded as a topic is a broadcast based channel. If you set the prefetch to one then you client would only be sent one message from the sever, then each time your onMessage callback completes a new message would be dispatched as the client would ack that message, that is if the session is in Auto Ack mode.
Refer to the NMS configuration page for all the options:
http://activemq.apache.org/nms/configuring.html
Regards
Tim.
FuseSource.com