I was able to consume multiple messages that are sent by multiple producers to the same exchange with different routing key using the above code and was able to insert each message to database.
But this will consume too much of resources as messages will be inserted into DB one after the other. So I decided to go for batch insert and I found I can set BasicQos
After setting the message limit to 10 in BasicQos, my expectation is the Console.WriteLine must write 10 messages, but it is not as expected.
My expectation is to consume N number messages from the queue and do bulk insert and on successful send ACK else No ACK
Here is the piece of code I use.
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_A");
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_B");
channel.BasicQos(0, 10, false);
var consumer = new EventingBasicConsumer(channel);
channel.BasicConsume(queue: "queueName", noAck: false, consumer: consumer);
consumer.Received += (model, ea) =>
{
try
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
// Insert into Database
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
Console.WriteLine(" Recevier Ack " + ea.DeliveryTag);
}
catch (Exception e)
{
channel.BasicNack(deliveryTag: ea.DeliveryTag, multiple: false, requeue: true);
Console.WriteLine(" Recevier No Ack " + ea.DeliveryTag);
}
};
Console.ReadLine();
}
}
BasicQos = 10 means that the client fetch only 10 messages at time, but when you consume it you will see always one message a time.
Read here: https://www.rabbitmq.com/consumer-prefetch.html
AMQP specifies the basic.qos method to allow you to limit the number
of unacknowledged messages on a channel (or connection) when consuming
(aka "prefetch count").
for your scope you have to download the messages, put it inside a temporary list and then insert into the DB.
an then you can use:
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: true);
void basicAck()
Parameters:
deliveryTag - the tag from the received
AMQP.Basic.GetOk or AMQP.Basic.Deliver
multiple - true to acknowledge
all messages up to and including the supplied delivery tag; false to
acknowledge just the supplied delivery tag.
Example
final List<String> myMessagges = new ArrayList<String>();
channel.basicConsume("my_queue", false, new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
myMessagges.add(new String(body));
System.out.println("Received...");
if (myMessagges.size() >= 10) {
System.out.println("insert into DB...");
channel.basicAck(envelope.getDeliveryTag(), true);
myMessagges.clear();
}
}
});
Batch size based consumption can be done using the channel.basicQos().
Channel channel = connection.createChannel();
channel.basicQos(10);
It specifies the maximum no of messages to be fetched without sending ACK for each.
Use the DefaultConsumer class and override its methods.
Consumer batchConsumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
}
#Override
public void handleCancelOk(String consumerTag) {
}
};
Consume 10 messages using channel.basicConsume()
channel.basicConsume(QUEUE_NAME, false, batchConsumer);
When channel.basicConsume() is called it will fetch a batch of 10 messages. 'false' is set to disable auto ack, and ACK to be sent only once after consuming entire batch.
channel.basicAck(getLastMessageEnvelope().getDeliveryTag(), true);
Here 'true' means we are sending ACK for multiple messages.
Detailed explanation can be found in
RabbitMQ Batch Consumption
Related
Is it possible to set the delivery tag on the message that I publish in my publisher client code? Is it possible to get the delivery tag for the message that I publish in my publisher client?
Here is my situation:
1) I am creating a job (job a) that creates many messages that might take a long time to process. Multiple jobs can be running at one time.
2) I am using a single queue
3) I am using a single receiver
4) I want the ability to "delete" all messages for job a.
If I knew what the delivery tag is then I can just ack each of the message to "delete" just the ones I want from the queue.
Thanks in advance.
You can set custom headers for every message in the following way:
Publisher:
byte[] messageBodyBytes = System.Text.Encoding.UTF8.GetBytes("Hello, world!");
IBasicProperties props = model.CreateBasicProperties();
props.ContentType = "text/plain";
props.DeliveryMode = 2;
props.Headers = new Dictionary<string, object>();
props.Headers.Add("header", value);
model.BasicPublish(exchangeName,
routingKey, props,
messageBodyBytes);
Reference: https://www.rabbitmq.com/dotnet-api-guide.html#publishing
At the consumer you have access to the BasicProperties.
Consumer:
public override void HandleBasicDeliver(string consumerTag, ulong deliveryTag, bool redelivered, string exchange, string routingKey, IBasicProperties properties, byte[] body)
{
header_value = // Read the value from the properites variable
if (header_value matches) {
// Reject or delete the message
_channel.BasicReject(deliveryTag, false);
}
else {
// Accept the message and do your processing
_channel.BasicAck(deliveryTag, false);
}
}
Reference: https://www.tutlane.com/tutorial/rabbitmq/csharp-read-messages-from-rabbitmq-queue
I am using RabbitMQ with pika which is a python client. I am not that fluent with C# or .NET, but I hope this helps!
In manual-acknowledgement mode (with noAck=false), messages may be acknowledged after being received and processed. Using IBasicConsumer, the .NET/C# API Guide suggests that this may be done, one-by-one, when a message is consumed:
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (_, e) =>
{
// process the message...
channel.BasicAck(e.DeliveryTag, false);
};
string consumerTag = channel.BasicConsume(queueName, false, consumer);
In my particular scenario, however, I need to consume several messages before I'm able to process them. And, in order to retain durability, I would have to acknowledge previous messages only later on (outside the scope of when they were received). These acknowledgements might also happen in a different order from the order in which messages are received.
var consumer = new EventingBasicConsumer(channel);
var buffer = new List<BasicDeliverEventArgs>();
consumer.Received += (_, e) =>
{
buffer.Add(e);
if (TryProcess(buffer, out IList<BasicDeliverEventArgs> subset))
{
foreach (var p in subset)
{
channel.BasicAck(p.DeliveryTag, false);
buffer.Remove(p);
}
}
};
string consumerTag = channel.BasicConsume(queueName, false, consumer);
When I try this I only receive a few messages and then something blocks, which leads me to thinking that RabbitMQ does not support this. I've also used the channel's BasicQos method to bump up the prefetchCount to hundreds, but the number of messages I receive is still only 4 or 5.
Does RabbitMQ support out-of-order delivery acknowledgements? And if not, is there a better technique I can apply that uses RabbitMQ to provide the same durability?
RabbitMQ 3.5 now supports message priority;
However, I am unable to build a working example. I've placed my code below. It includes the output that I expect and the output I actually. I'd be interested in more documentation, and/or a working example.
So my question in short: How do I get message priority to work in Rabbit 3.5.0.0?
Publisher:
using System;
using RabbitMQ.Client;
using System.Text;
using System.Collections.Generic;
class Publisher
{
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
IDictionary <String , Object> args = new Dictionary<String,Object>() ;
args.Add(" x-max-priority ", 10);
channel.QueueDeclare("task_queue1", true, false, true, args);
for (int i = 1 ; i<=10; i++ )
{
var message = "Message";
var body = Encoding.UTF8.GetBytes(message + " " + i);
var properties = channel.CreateBasicProperties();
properties.SetPersistent(true);
properties.Priority = Convert.ToByte(i);
channel.BasicPublish("", "task_queue1", properties, body);
}
}
}
}
}
Consumer:
using System;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System.Text;
using System.Threading;
using System.Collections.Generic;
namespace Consumer
{
class Worker
{
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
IDictionary<String, Object> args = new Dictionary<String, Object>();
channel.BasicQos(0, 1, false);
var consumer = new QueueingBasicConsumer(channel);
IDictionary<string, object> consumerArgs = new Dictionary<string, object>();
channel.BasicConsume( "task_queue1", false, "", args, consumer);
Console.WriteLine(" [*] Waiting for messages. " +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
channel.BasicAck(ea.DeliveryTag, false);
}
}
}
}
}
}
Actual output:
[*] Waiting for messages. To exit press CTRL+C
[x] Received Message 1
[x] Received Message 2
[x] Received Message 3
[x] Received Message 4
[x] Received Message 5
[x] Received Message 6
[x] Received Message 7
[x] Received Message 8
[x] Received Message 9
[x] Received Message 10
Expected output:
[*] Waiting for messages. To exit press CTRL+C
[x] Received Message 10
[x] Received Message 9
[x] Received Message 8
[x] Received Message 7
[x] Received Message 6
[x] Received Message 5
[x] Received Message 4
[x] Received Message 3
[x] Received Message 2
[x] Received Message 1
UPDATE #1.
I found an example in Java here. However it's the Rabbit 3.4.x.x. addin that was incorporated into 3.5.
The only difference I can see is that they express the priority as an int and mine is a byte. But I feel like that's a red herring. I'm at a bit of a loss here.
Well I solved it.
It was a dumb mistake.
I wrote:
args.Add(" x-max-priority ", 10);
It should have been
args.Add("x-max-priority", 10);
I'll leave this up so other people can have a working example of Rabbitmq 3.5's Priority Queues in C#.
A similar RabbitMq Priority Queue Implementation in Node JS
Install amqplib
In order to test, we are required to have amqplib installed
npm install amqplib
Publisher (send.js)
#!/usr/bin/env node
var amqp = require('amqplib/callback_api');
function bail(err, conn) {
console.error(err);
if (conn) conn.close(function() { process.exit(1); });
}
function on_connect(err, conn) {
if (err !== null) return bail(err);
// name of queue
var q = 'hello';
var msg = 'Hello World!';
var priorityValue = 0;
function on_channel_open(err, ch) {
if (err !== null) return bail(err, conn);
// maxPriority : max priority value supported by queue
ch.assertQueue(q, {durable: false, maxPriority: 10}, function(err, ok) {
if (err !== null) return bail(err, conn);
for(var index=1; index<=100; index++) {
priorityValue = Math.floor((Math.random() * 10));
msg = 'Hello World!' + ' ' + index + ' ' + priorityValue;
ch.publish('', q, new Buffer(msg), {priority: priorityValue});
console.log(" [x] Sent '%s'", msg);
}
ch.close(function() { conn.close(); });
});
}
conn.createChannel(on_channel_open);
}
amqp.connect(on_connect);
Subscriber (receive.js)
#!/usr/bin/env node
var amqp = require('amqplib/callback_api');
function bail(err, conn) {
console.error(err);
if (conn) conn.close(function() { process.exit(1); });
}
function on_connect(err, conn) {
if (err !== null) return bail(err);
process.once('SIGINT', function() { conn.close(); });
var q = 'hello';
function on_channel_open(err, ch) {
ch.assertQueue(q, {durable: false, maxPriority: 10}, function(err, ok) {
if (err !== null) return bail(err, conn);
ch.consume(q, function(msg) { // message callback
console.log(" [x] Received '%s'", msg.content.toString());
}, {noAck: true}, function(_consumeOk) { // consume callback
console.log(' [*] Waiting for messages. To exit press CTRL+C');
});
});
}
conn.createChannel(on_channel_open);
}
amqp.connect(on_connect);
Run:
node send.js
It will create a queue named 'hello' and will flood it with '1000' sample messages using default AMQP exchange.
node receive.js
It will act as a consumer to subscribe to messages waiting in the queue.
Another possibility (for future searchers)
The "Push" method of message delivery doesn't seem to respect Priority.
http://rabbitmq.docs.pivotal.io/35/rabbit-web-docs/dotnet-api-guide.html.html
The below is a quote from the URL above. I've bolded the important part.
Retrieving Messages By Subscription ("push API")
Another way to receive messages is to set up a subscription using the IBasicConsumer interface. The messages will then be delivered automatically as they arrive, rather than having to be requested proactively. One way to implement a consumer is to use the convenience class EventingBasicConsumer, which dispatches deliveries and other consumer lifecycle events as C# events:
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (ch, ea) =>
{
var body = ea.Body;
// ... process the message
ch.BasicAck(ea.DeliveryTag, false);
};
String consumerTag = channel.BasicConsume(queueName, false, consumer);
By changing to the "pull" method, Priority seems to be respected. However, in the quote below (from the same url above), it looks like there is a trade-off (that I've bolded)
Fetching Individual Messages ("pull API")
To retrieve individual messages, use IModel.BasicGet. The returned value is an instance of BasicGetResult, from which the header information (properties) and message body can be extracted:
bool noAck = false;
BasicGetResult result = channel.BasicGet(queueName, noAck);
if (result == null) {
// No message available at this time.
} else {
IBasicProperties props = result.BasicProperties;
byte[] body = result.Body;
...
Since noAck = false above, you must also call IModel.BasicAck to acknowledge that you have successfully received and processed the message:
...
// acknowledge receipt of the message
channel.BasicAck(result.DeliveryTag, false);
}
Note that fetching messages using this API is relatively inefficient. If you'd prefer RabbitMQ to push messages to the client, see the next section.
(The "next" section in this case takes you to the "push" method at the top of this post)
MessageType: "PublishX"
Consumers:
Type1ConsumerX
Type2ConsumerX
Type3ConsumerX
All of the consumers must catch messages immediately, but consume synchronously inside themselves..
For example there are 100 "PublishX" messages in the queue. Type1ConsumerX consumed 30 messages (synchronously), Type2ConsumerX consumed 50 messages(synchronously) , Type3ConsumerX consumed 100 messages(synchronously).
How can I know the message is consumed by "all type of consumers" ?
Could RabbitMQ/MassTransit PUSH messages to consumers?
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
If I've understood the question correctly you just need to set up a basic pub/sub pattern. This will allow you to deliver the same message to multiple consumers.
Example publisher:
public static void PublishMessageToFanout()
{
var factory = new ConnectionFactory { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
var message = new Message { Text = "This is a message to send" };
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.BasicPublish("messages", string.Empty, null, body);
}
}
Example consumers:
SubscribeToMessages("sms-messages", (s) => Console.WriteLine("SMS Message: {0}", s));
SubscribeToMessages("email-messages", (s) => Console.WriteLine("Email Message: {0}", s));
public static void SubscribeToMessages(string queueName, Action<string> messageAction)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
channel.QueueDeclare(queueName, true, false, false, null);
channel.QueueBind(queueName, "messages", string.Empty);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(queueName, true, consumer);
while (true)
{
var ea = consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
messageAction(message);
}
}
}
If you run the SubscribeToMessages loops in separate processes or console apps you'll see that they both print out the message whenever you call the PublishMessageToFanout. You'll also see that both queues exist in RabbitMQ Management under Queues.
Regarding the MassTransit part of your question
RabbitMQ/MassTransit PUSH messages to consumers?
Yes, MassTransit publishes messages to the bus, and then a consumer processes them
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Don't know if there is a feature for this, you could write your own but you would have to be very careful about losing the messages.
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
Yes, multiple consumers can consume the same type of message.
I've written a simple hello world app that shows the basics - http://nodogmablog.bryanhogan.net/2015/04/mass-transit-with-rabbitmq-hello-world/
The basic RabbitMQ tutorial gives an example of how to retrieve messages continuously from a queue:
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
}
What I want to do is retrieve all messages which have been placed onto the queue and then stop.
Here are two examples that would solve my problem
If I start my code at 1pm, I want to process all the messages that have been placed on the queue before 1pm.
OR
If I start my code at 13:00:00, and it takes 10 seconds for my code to run, I don't mind if it includes messages placed on the queue between 13:00:00 and 13:00:10, as long as it stops as soon as the queue is empty.
I realize that I can probably put a time stamp in my message and check for that, or I could fiddle with timeout values, but I was wondering if there's any built in way to do this properly.
Thanks in advance.
From the comments it seems that RabbitMQ is not meant for batch processing, so it hasn't been designed for this purpose.
I've also noticed that, when testing the DequeueNoWait method, or attempting to Dequeue with a zero timeout, they didn't work at all, and simply returned null.
The following solution uses QueueDeclare to get a count of existing messages and doesn't require a time stamp or a hacky timeout:
var factory = new ConnectionFactory { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
var queueDeclareResponse = channel.QueueDeclare(Constants.QueueName, false, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(Constants.QueueName, true, consumer);
Console.WriteLine(" [*] Processing existing messages.");
for (int i = 0; i < queueDeclareResponse.MessageCount; i++)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
Console.WriteLine("Finished processing {0} messages.", queueDeclareResponse.MessageCount);
Console.ReadLine();
}
}
What you can do:
1.) First add a timestamp to the message object
2.) Reject the message if the timestamp is not valid. It is an integrated feature of rabbitmq
Reference