In manual-acknowledgement mode (with noAck=false), messages may be acknowledged after being received and processed. Using IBasicConsumer, the .NET/C# API Guide suggests that this may be done, one-by-one, when a message is consumed:
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (_, e) =>
{
// process the message...
channel.BasicAck(e.DeliveryTag, false);
};
string consumerTag = channel.BasicConsume(queueName, false, consumer);
In my particular scenario, however, I need to consume several messages before I'm able to process them. And, in order to retain durability, I would have to acknowledge previous messages only later on (outside the scope of when they were received). These acknowledgements might also happen in a different order from the order in which messages are received.
var consumer = new EventingBasicConsumer(channel);
var buffer = new List<BasicDeliverEventArgs>();
consumer.Received += (_, e) =>
{
buffer.Add(e);
if (TryProcess(buffer, out IList<BasicDeliverEventArgs> subset))
{
foreach (var p in subset)
{
channel.BasicAck(p.DeliveryTag, false);
buffer.Remove(p);
}
}
};
string consumerTag = channel.BasicConsume(queueName, false, consumer);
When I try this I only receive a few messages and then something blocks, which leads me to thinking that RabbitMQ does not support this. I've also used the channel's BasicQos method to bump up the prefetchCount to hundreds, but the number of messages I receive is still only 4 or 5.
Does RabbitMQ support out-of-order delivery acknowledgements? And if not, is there a better technique I can apply that uses RabbitMQ to provide the same durability?
Related
I am using Azure.Messaging.ServiceBus nuget package to work with Azure service bus. We have created a topic and a subscription. The subscription has 100+ messages. We want to read all the message and continue to read message as they arrive.
Microsoft.Azure.ServiceBus package (deprecated now) provided RegisterMessageHandler which use to process every incoming message. I am not able to find similar option under Azure.Messaging.ServiceBus nuget package.
I am able to read one message at a time but I have to call await receiver.ReceiveMessageAsync(); every time manually.
To receive multiple messages (a batch), you should use ServiceBusReceiver.ReceiveMessagesAsync() (not plural, not singular 'message'). This method will return whatever number of messages it can send back. To ensure you retrieve all 100+ messages, you'll need to loop until no messages are available.
If you'd like to use a processor, that's also available in the new SDK. See my answer to a similar question here.
As suggested by #gaurav Mantri, I used ServiceBusProcessor class to implement event based model for processing messages
public async Task ReceiveAll()
{
string connectionString = "Endpoint=sb://sb-test-today.servicebus.windows.net/;SharedAccessKeyName=manage;SharedAccessKey=8e+6SWp3skB3Aedsadsadasdwz5DU=;";
string topicName = "topicone";
string subscriptionName = "subone";
await using var client = new ServiceBusClient(connectionString, new ServiceBusClientOptions
{
TransportType = ServiceBusTransportType.AmqpWebSockets
});
var options = new ServiceBusProcessorOptions
{
// By default or when AutoCompleteMessages is set to true, the processor will complete the message after executing the message handler
// Set AutoCompleteMessages to false to [settle messages](https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement#peeklock) on your own.
// In both cases, if the message handler throws an exception without settling the message, the processor will abandon the message.
AutoCompleteMessages = false,
// I can also allow for multi-threading
MaxConcurrentCalls = 1
};
await using ServiceBusProcessor processor = client.CreateProcessor(topicName, subscriptionName, options);
processor.ProcessMessageAsync += MessageHandler;
processor.ProcessErrorAsync += ErrorHandler;
await processor.StartProcessingAsync();
Console.ReadKey();
}
public async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine(body);
// we can evaluate application logic and use that to determine how to settle the message.
await args.CompleteMessageAsync(args.Message);
}
public Task ErrorHandler(ProcessErrorEventArgs args)
{
// the error source tells me at what point in the processing an error occurred
Console.WriteLine(args.ErrorSource);
// the fully qualified namespace is available
Console.WriteLine(args.FullyQualifiedNamespace);
// as well as the entity path
Console.WriteLine(args.EntityPath);
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
I want to create method:
bool TryPullMessage(string queue, out T message, TimeSpan? timeout = null);
which will try to pull one message from queue (I configured Qos for 1 message at a time) and will fail after timeout. For implementation I have two choises:
To use manual request in loop: model.BasicGet(queue, ...)
To use consumer: new EvenBasicConsumer(model)
However, I don't want to use first approach because it creates unlimited traffic and resource consumption. For the second approach I tried many things like this:
var parent = new EventWaitHandle(false, EventResetMode.AutoReset);
var child = new EventWaitHandle(false, EventResetMode.AutoReset);
var consumer = new EventingBasicConsumer(_model);
consumer.Received += (sender, args) =>
{
//save message localy
parent.Set();
child.WaitOne();
};
_model.BasicConsume(queue, true, consumer);
try
{
parent.WaitOne(timeout.Value);
}
finally
{
_model.BasicCancel(consumer.ConsumerTag);
child.Set();
}
But without success, it just keeps consuming them. Is it possible to use RabbitMQ pushes for one single message?
I was able to consume multiple messages that are sent by multiple producers to the same exchange with different routing key using the above code and was able to insert each message to database.
But this will consume too much of resources as messages will be inserted into DB one after the other. So I decided to go for batch insert and I found I can set BasicQos
After setting the message limit to 10 in BasicQos, my expectation is the Console.WriteLine must write 10 messages, but it is not as expected.
My expectation is to consume N number messages from the queue and do bulk insert and on successful send ACK else No ACK
Here is the piece of code I use.
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_A");
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_B");
channel.BasicQos(0, 10, false);
var consumer = new EventingBasicConsumer(channel);
channel.BasicConsume(queue: "queueName", noAck: false, consumer: consumer);
consumer.Received += (model, ea) =>
{
try
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
// Insert into Database
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
Console.WriteLine(" Recevier Ack " + ea.DeliveryTag);
}
catch (Exception e)
{
channel.BasicNack(deliveryTag: ea.DeliveryTag, multiple: false, requeue: true);
Console.WriteLine(" Recevier No Ack " + ea.DeliveryTag);
}
};
Console.ReadLine();
}
}
BasicQos = 10 means that the client fetch only 10 messages at time, but when you consume it you will see always one message a time.
Read here: https://www.rabbitmq.com/consumer-prefetch.html
AMQP specifies the basic.qos method to allow you to limit the number
of unacknowledged messages on a channel (or connection) when consuming
(aka "prefetch count").
for your scope you have to download the messages, put it inside a temporary list and then insert into the DB.
an then you can use:
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: true);
void basicAck()
Parameters:
deliveryTag - the tag from the received
AMQP.Basic.GetOk or AMQP.Basic.Deliver
multiple - true to acknowledge
all messages up to and including the supplied delivery tag; false to
acknowledge just the supplied delivery tag.
Example
final List<String> myMessagges = new ArrayList<String>();
channel.basicConsume("my_queue", false, new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
myMessagges.add(new String(body));
System.out.println("Received...");
if (myMessagges.size() >= 10) {
System.out.println("insert into DB...");
channel.basicAck(envelope.getDeliveryTag(), true);
myMessagges.clear();
}
}
});
Batch size based consumption can be done using the channel.basicQos().
Channel channel = connection.createChannel();
channel.basicQos(10);
It specifies the maximum no of messages to be fetched without sending ACK for each.
Use the DefaultConsumer class and override its methods.
Consumer batchConsumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
}
#Override
public void handleCancelOk(String consumerTag) {
}
};
Consume 10 messages using channel.basicConsume()
channel.basicConsume(QUEUE_NAME, false, batchConsumer);
When channel.basicConsume() is called it will fetch a batch of 10 messages. 'false' is set to disable auto ack, and ACK to be sent only once after consuming entire batch.
channel.basicAck(getLastMessageEnvelope().getDeliveryTag(), true);
Here 'true' means we are sending ACK for multiple messages.
Detailed explanation can be found in
RabbitMQ Batch Consumption
I'm having an issue with my Rabbit queues that is currently only reacting to the first message in queue, after that any other messages being pushed are being ignored.
I start with instantiating the connection and declaring the queue in my IQueueConnectionProvider:
var connectionFactory = new ConnectionFactory() { HostName = hostName };
var connection = _connectionFactory.CreateConnection();
var channel = connection.CreateModel();
That IQueueConnectionProvider is then used in my IQueueListener as a dependency with just one method:
public void ListenToQueue(string queue)
{
var channel = _queueConnectionProvider.GetQueue();
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
string path = #"d:\debug.log.txt";
File.AppendAllLines(path, new List<string>() {"MESSAGE RECEIVED", Environment.NewLine });
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
channel.BasicAck(ea.DeliveryTag, false);
};
channel.BasicConsume(queue, true, consumer);
}
My log file ends up being just one line "MESSAGE RECEIVED", however I can see in the Rabbit ui interface that my other services are pushing the messages to that queue just fine.
Is there something I'm missing here?
This was a dumb mistake... yet again.
channel.BasicConsume(queue, false, consumer);
This was what I needed. I want to manually acknowledge my messages, therefore noAck needs to be false;
The code works fine! Have tested with my queue, and was able to get "MESSAGE RECEIVED" 9 times in the log file; since I had 9 messages in my queue.
I tried without this line of code, and it worked fine
var channel = _queueConnectionProvider.GetQueue();
MessageType: "PublishX"
Consumers:
Type1ConsumerX
Type2ConsumerX
Type3ConsumerX
All of the consumers must catch messages immediately, but consume synchronously inside themselves..
For example there are 100 "PublishX" messages in the queue. Type1ConsumerX consumed 30 messages (synchronously), Type2ConsumerX consumed 50 messages(synchronously) , Type3ConsumerX consumed 100 messages(synchronously).
How can I know the message is consumed by "all type of consumers" ?
Could RabbitMQ/MassTransit PUSH messages to consumers?
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
If I've understood the question correctly you just need to set up a basic pub/sub pattern. This will allow you to deliver the same message to multiple consumers.
Example publisher:
public static void PublishMessageToFanout()
{
var factory = new ConnectionFactory { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
var message = new Message { Text = "This is a message to send" };
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.BasicPublish("messages", string.Empty, null, body);
}
}
Example consumers:
SubscribeToMessages("sms-messages", (s) => Console.WriteLine("SMS Message: {0}", s));
SubscribeToMessages("email-messages", (s) => Console.WriteLine("Email Message: {0}", s));
public static void SubscribeToMessages(string queueName, Action<string> messageAction)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
channel.QueueDeclare(queueName, true, false, false, null);
channel.QueueBind(queueName, "messages", string.Empty);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(queueName, true, consumer);
while (true)
{
var ea = consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
messageAction(message);
}
}
}
If you run the SubscribeToMessages loops in separate processes or console apps you'll see that they both print out the message whenever you call the PublishMessageToFanout. You'll also see that both queues exist in RabbitMQ Management under Queues.
Regarding the MassTransit part of your question
RabbitMQ/MassTransit PUSH messages to consumers?
Yes, MassTransit publishes messages to the bus, and then a consumer processes them
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Don't know if there is a feature for this, you could write your own but you would have to be very careful about losing the messages.
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
Yes, multiple consumers can consume the same type of message.
I've written a simple hello world app that shows the basics - http://nodogmablog.bryanhogan.net/2015/04/mass-transit-with-rabbitmq-hello-world/