Develop Reference

Azure Service Bus Receive Messages continuously when ever new message placed in web application [duplicate]

I am using Azure.Messaging.ServiceBus nuget package to work with Azure service bus. We have created a topic and a subscription. The subscription has 100+ messages. We want to read all the message and continue to read message as they arrive.
Microsoft.Azure.ServiceBus package (deprecated now) provided RegisterMessageHandler which use to process every incoming message. I am not able to find similar option under Azure.Messaging.ServiceBus nuget package.
I am able to read one message at a time but I have to call await receiver.ReceiveMessageAsync(); every time manually.

To receive multiple messages (a batch), you should use ServiceBusReceiver.ReceiveMessagesAsync() (not plural, not singular 'message'). This method will return whatever number of messages it can send back. To ensure you retrieve all 100+ messages, you'll need to loop until no messages are available.
If you'd like to use a processor, that's also available in the new SDK. See my answer to a similar question here.

As suggested by #gaurav Mantri, I used ServiceBusProcessor class to implement event based model for processing messages
public async Task ReceiveAll()
{
string connectionString = "Endpoint=sb://sb-test-today.servicebus.windows.net/;SharedAccessKeyName=manage;SharedAccessKey=8e+6SWp3skB3Aedsadsadasdwz5DU=;";
string topicName = "topicone";
string subscriptionName = "subone";
await using var client = new ServiceBusClient(connectionString, new ServiceBusClientOptions
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
var options = new ServiceBusProcessorOptions
{
// By default or when AutoCompleteMessages is set to true, the processor will complete the message after executing the message handler
// Set AutoCompleteMessages to false to [settle messages](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement#peeklock) on your own.
// In both cases, if the message handler throws an exception without settling the message, the processor will abandon the message.
AutoCompleteMessages = false,
// I can also allow for multi-threading
MaxConcurrentCalls = 1
};
await using ServiceBusProcessor processor = client.CreateProcessor(topicName, subscriptionName, options);
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.ReadKey();
}
public async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine(body);
// we can evaluate application logic and use that to determine how to settle the message.
await args.CompleteMessageAsync(args.Message);
}
public Task ErrorHandler(ProcessErrorEventArgs args)
{
// the error source tells me at what point in the processing an error occurred
Console.WriteLine(args.ErrorSource);
// the fully qualified namespace is available
Console.WriteLine(args.FullyQualifiedNamespace);
// as well as the entity path
Console.WriteLine(args.EntityPath);
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}

Out-of-order delivery acknowledgements in RabbitMQ

In manual-acknowledgement mode (with noAck=false), messages may be acknowledged after being received and processed. Using IBasicConsumer, the .NET/C# API Guide suggests that this may be done, one-by-one, when a message is consumed:
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (_, e) =>
{
// process the message...
channel.BasicAck(e.DeliveryTag, false);
};
string consumerTag = channel.BasicConsume(queueName, false, consumer);
In my particular scenario, however, I need to consume several messages before I'm able to process them. And, in order to retain durability, I would have to acknowledge previous messages only later on (outside the scope of when they were received). These acknowledgements might also happen in a different order from the order in which messages are received.
var consumer = new EventingBasicConsumer(channel);
var buffer = new List<BasicDeliverEventArgs>();
consumer.Received += (_, e) =>
{
buffer.Add(e);
if (TryProcess(buffer, out IList<BasicDeliverEventArgs> subset))
{
foreach (var p in subset)
{
channel.BasicAck(p.DeliveryTag, false);
buffer.Remove(p);
}
}
};
string consumerTag = channel.BasicConsume(queueName, false, consumer);
When I try this I only receive a few messages and then something blocks, which leads me to thinking that RabbitMQ does not support this. I've also used the channel's BasicQos method to bump up the prefetchCount to hundreds, but the number of messages I receive is still only 4 or 5.
Does RabbitMQ support out-of-order delivery acknowledgements? And if not, is there a better technique I can apply that uses RabbitMQ to provide the same durability?

RabbitMQ only listens to the first message on a queue

I'm having an issue with my Rabbit queues that is currently only reacting to the first message in queue, after that any other messages being pushed are being ignored.
I start with instantiating the connection and declaring the queue in my IQueueConnectionProvider:
var connectionFactory = new ConnectionFactory() { HostName = hostName };
var connection = _connectionFactory.CreateConnection();
var channel = connection.CreateModel();
That IQueueConnectionProvider is then used in my IQueueListener as a dependency with just one method:
public void ListenToQueue(string queue)
{
var channel = _queueConnectionProvider.GetQueue();
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
string path = #"d:\debug.log.txt";
File.AppendAllLines(path, new List<string>() {"MESSAGE RECEIVED", Environment.NewLine });
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
channel.BasicAck(ea.DeliveryTag, false);
};
channel.BasicConsume(queue, true, consumer);
}
My log file ends up being just one line "MESSAGE RECEIVED", however I can see in the Rabbit ui interface that my other services are pushing the messages to that queue just fine.
Is there something I'm missing here?

This was a dumb mistake... yet again.
channel.BasicConsume(queue, false, consumer);
This was what I needed. I want to manually acknowledge my messages, therefore noAck needs to be false;

The code works fine! Have tested with my queue, and was able to get "MESSAGE RECEIVED" 9 times in the log file; since I had 9 messages in my queue.
I tried without this line of code, and it worked fine
var channel = _queueConnectionProvider.GetQueue();

Azure service bus queue Message deadlettered after Message.Abandon

I am trying out Azure Service Bus queue. I have the below code:
Queue send:
string strConnectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
var namespaceManager = NamespaceManager.CreateFromConnectionString(strConnectionString);
if (!namespaceManager.QueueExists("Test"))
{
QueueDescription qD = new QueueDescription("Test");
qD.DefaultMessageTimeToLive = new TimeSpan(05, 00, 00);
qD.LockDuration = new TimeSpan(00, 02, 30);
qD.MaxSizeInMegabytes = 5120;
namespaceManager.CreateQueue(qD);
}
if (namespaceManager.QueueExists("Test"))
{
QueueClient client = QueueClient.CreateFromConnectionString(strConnectionString, "Test", ReceiveMode.PeekLock);
var qMessage = Console.ReadLine();
using (MemoryStream strm = new MemoryStream(Encoding.UTF8.GetBytes(qMessage)))
{
BrokeredMessage bMsg = new BrokeredMessage(strm);
bMsg.MessageId = Guid.NewGuid().ToString();
bMsg.TimeToLive = new TimeSpan(05, 00, 00);
client.Send(bMsg);
Console.WriteLine("Message sent");
}
}
Console.ReadLine();
The receive code:
string strConnectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
var namespaceManager = NamespaceManager.CreateFromConnectionString(strConnectionString);
if (namespaceManager.QueueExists("Test"))
{
QueueClient client = QueueClient.CreateFromConnectionString(strConnectionString, "Test",ReceiveMode.PeekLock);
if (client != null)
{
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromSeconds(31);
client.OnMessage((message) =>
{
Console.WriteLine(message.State.ToString());
Console.WriteLine("Message Id: " + message.MessageId);
Stream stream = message.GetBody<Stream>();
StreamReader reader = new StreamReader(stream);
Console.WriteLine("Message: " + reader.ReadToEnd());
Console.WriteLine("***************");
message.Abandon();
});
Console.ReadLine();
}
}
I see that whenever I call Abandon, the message is getting DeadLettered. My assumption was that it should get Active and can be picked up by another client.

Your understanding of BrokeredMessage.Abandon Api is correct. It is intended to abandon the peek-lock acquired on the message (but NOT abandon the message itself) and hence, makes it available for other receivers to pick the Message up.
Here's how we envisioned different states of a peek-lock'ed message:
Basics first
The 'Why': If Customers need Competing-Consumer (Job-Queue) semantics - where they need multiple workers to simultaneously process different messages from a Queue with Exactly-Once guarantee - then they use the ReceiveMode.PeekLock. In this model, each worker (the queue receiver) needs a way to communicate the Progress of its Current message (Job) to other workers. Hence, brokeredMessage provides 4 functions to express the states.
The 'What':
if a message is successfully processed by the current Worker - call BrokeredMessage.Complete()
if the BrokeredMessage cannot be processed by the current worker, and want the processing to be retried on another Worker - then, Abandon the message. But, the catch here is: lets say, there are 2 workers and each of them thinks that the other one can process this message and calls Abandon - soon they will end up in an Infinite loop of retry'ing to process just that message! So, to avoid this situation, we provided a Configuration called MaxDeliveryCount on QueueDescription. This setting guards the limit on the number of times the message is delivered from the Queue to receiver. In the above example, Each time you received (and abandoned) the message, the 'deliveryCount' on the ServiceBus service is incremented. When it reaches 10 - the message has hit max no. of deliveries and hence, will be deadlettered.
if the current receiver (worker) knows for sure, that, this message cannot be processed, BrokeredMessage.DeadLetter(). The goal here is to let the consuming application Audit the dead-lettered messages regularly.
if the current receiver (worker) cannot process this message, but, knows that this message can be processed at a later point of time BrokeredMessage.Defer().
HTH!
Sree

What is an elegant way of halting consumption of messages gracefully in the C# client for RabbitMQ?

I am setting up a standard standalone thread listening to RabbitMQ in C#. Suppose the method for listening in the thread looks like this:
public void Listen()
{
using (var channel = connection.CreateModel())
{
var consumer = SetupQueues(channel);
while (true)
{
var ea = consumer.Queue.Dequeue(); // blocking call
handler.HandleMessage(channel, ea);
}
}
}
What is an elegant way of halting consumption of messages gracefully in the C# client for RabbitMQ? Keep in mind I have found nothing of use in the RabbitMQ examples/docs or these SO questions:
How to stop consuming message from selective queue - RabbitMQ
How to pause and resume consumption gracefully in rabbitmq, pika python
What is the best way to safely end a java application with running RabbitMQ consumers
The issue here is consumer.Queue.Dequeue() is a blocking call. I have tried these options:
Calling channel.BasicCancel(string tag). This causes a System.IO.EndOfStreamException in the blocking call. I do not want to use this exception as part of the control flow for obvious reasons.
Calling consumer.Queue.Dequeue(int millisecondsTimeout, out T result) and checking a flag in between loop iterations. This can work but seems hacky.
I want to let the thread exit gracefully and clean up any unmanaged resources I might have, so no thread aborting, etc.
Any help is appreciated. Thanks

The DeQueue with the timeout & flag is the way to do it. It's a very common pattern, and is why many blocking calls are provided with timeout-enabled versions.
Alternately, throwing a (known) exception isn't necessarily a bad thing for control flow. Gracefully shutting down could mean actually trapping the exception, commenting "this is thrown when requesting the channel shuts down", and then returning cleanly. This is how part of TPL works with the CancellationToken.

The blocking methods are not property event-driven.
I don't why they suggest to use consumer.Queue.Dequeue();
Anyway, I usually don't use consumer.Queue.Dequeue();
I extend the default consumer, in this way:
class MyConsumer : DefaultBasicConsumer {
public MyConsumer(IModel model):base(model)
{
}
public override void HandleBasicDeliver(string consumerTag, ulong deliveryTag, bool redelivered, string exchange, string routingKey, IBasicProperties properties, byte[] body) {
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
class Program
{
static void Main(string[] args)
{
var factory = new ConnectionFactory() { Uri = "amqp://aa:bbb#lemur.cloudamqp.com/xxx" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new MyConsumer(channel);
String tag = channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
" any key to exit");
Console.ReadLine();
channel.BasicCancel(tag);
/*while (true)
{
/////// DON'T USE THIS
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}*/
}
}
}
}
In this way you don't have a blocking method, and you can release all the resources correctly.
EDIT
I think using ctrl+C to break a program is always wrong.