RabbitMQ 3.5 now supports message priority;
However, I am unable to build a working example. I've placed my code below. It includes the output that I expect and the output I actually. I'd be interested in more documentation, and/or a working example.
So my question in short: How do I get message priority to work in Rabbit 3.5.0.0?
Publisher:
using System;
using RabbitMQ.Client;
using System.Text;
using System.Collections.Generic;
class Publisher
{
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
IDictionary <String , Object> args = new Dictionary<String,Object>() ;
args.Add(" x-max-priority ", 10);
channel.QueueDeclare("task_queue1", true, false, true, args);
for (int i = 1 ; i<=10; i++ )
{
var message = "Message";
var body = Encoding.UTF8.GetBytes(message + " " + i);
var properties = channel.CreateBasicProperties();
properties.SetPersistent(true);
properties.Priority = Convert.ToByte(i);
channel.BasicPublish("", "task_queue1", properties, body);
}
}
}
}
}
Consumer:
using System;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System.Text;
using System.Threading;
using System.Collections.Generic;
namespace Consumer
{
class Worker
{
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
IDictionary<String, Object> args = new Dictionary<String, Object>();
channel.BasicQos(0, 1, false);
var consumer = new QueueingBasicConsumer(channel);
IDictionary<string, object> consumerArgs = new Dictionary<string, object>();
channel.BasicConsume( "task_queue1", false, "", args, consumer);
Console.WriteLine(" [*] Waiting for messages. " +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
channel.BasicAck(ea.DeliveryTag, false);
}
}
}
}
}
}
Actual output:
[*] Waiting for messages. To exit press CTRL+C
[x] Received Message 1
[x] Received Message 2
[x] Received Message 3
[x] Received Message 4
[x] Received Message 5
[x] Received Message 6
[x] Received Message 7
[x] Received Message 8
[x] Received Message 9
[x] Received Message 10
Expected output:
[*] Waiting for messages. To exit press CTRL+C
[x] Received Message 10
[x] Received Message 9
[x] Received Message 8
[x] Received Message 7
[x] Received Message 6
[x] Received Message 5
[x] Received Message 4
[x] Received Message 3
[x] Received Message 2
[x] Received Message 1
UPDATE #1.
I found an example in Java here. However it's the Rabbit 3.4.x.x. addin that was incorporated into 3.5.
The only difference I can see is that they express the priority as an int and mine is a byte. But I feel like that's a red herring. I'm at a bit of a loss here.
Well I solved it.
It was a dumb mistake.
I wrote:
args.Add(" x-max-priority ", 10);
It should have been
args.Add("x-max-priority", 10);
I'll leave this up so other people can have a working example of Rabbitmq 3.5's Priority Queues in C#.
A similar RabbitMq Priority Queue Implementation in Node JS
Install amqplib
In order to test, we are required to have amqplib installed
npm install amqplib
Publisher (send.js)
#!/usr/bin/env node
var amqp = require('amqplib/callback_api');
function bail(err, conn) {
console.error(err);
if (conn) conn.close(function() { process.exit(1); });
}
function on_connect(err, conn) {
if (err !== null) return bail(err);
// name of queue
var q = 'hello';
var msg = 'Hello World!';
var priorityValue = 0;
function on_channel_open(err, ch) {
if (err !== null) return bail(err, conn);
// maxPriority : max priority value supported by queue
ch.assertQueue(q, {durable: false, maxPriority: 10}, function(err, ok) {
if (err !== null) return bail(err, conn);
for(var index=1; index<=100; index++) {
priorityValue = Math.floor((Math.random() * 10));
msg = 'Hello World!' + ' ' + index + ' ' + priorityValue;
ch.publish('', q, new Buffer(msg), {priority: priorityValue});
console.log(" [x] Sent '%s'", msg);
}
ch.close(function() { conn.close(); });
});
}
conn.createChannel(on_channel_open);
}
amqp.connect(on_connect);
Subscriber (receive.js)
#!/usr/bin/env node
var amqp = require('amqplib/callback_api');
function bail(err, conn) {
console.error(err);
if (conn) conn.close(function() { process.exit(1); });
}
function on_connect(err, conn) {
if (err !== null) return bail(err);
process.once('SIGINT', function() { conn.close(); });
var q = 'hello';
function on_channel_open(err, ch) {
ch.assertQueue(q, {durable: false, maxPriority: 10}, function(err, ok) {
if (err !== null) return bail(err, conn);
ch.consume(q, function(msg) { // message callback
console.log(" [x] Received '%s'", msg.content.toString());
}, {noAck: true}, function(_consumeOk) { // consume callback
console.log(' [*] Waiting for messages. To exit press CTRL+C');
});
});
}
conn.createChannel(on_channel_open);
}
amqp.connect(on_connect);
Run:
node send.js
It will create a queue named 'hello' and will flood it with '1000' sample messages using default AMQP exchange.
node receive.js
It will act as a consumer to subscribe to messages waiting in the queue.
Another possibility (for future searchers)
The "Push" method of message delivery doesn't seem to respect Priority.
http://rabbitmq.docs.pivotal.io/35/rabbit-web-docs/dotnet-api-guide.html.html
The below is a quote from the URL above. I've bolded the important part.
Retrieving Messages By Subscription ("push API")
Another way to receive messages is to set up a subscription using the IBasicConsumer interface. The messages will then be delivered automatically as they arrive, rather than having to be requested proactively. One way to implement a consumer is to use the convenience class EventingBasicConsumer, which dispatches deliveries and other consumer lifecycle events as C# events:
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (ch, ea) =>
{
var body = ea.Body;
// ... process the message
ch.BasicAck(ea.DeliveryTag, false);
};
String consumerTag = channel.BasicConsume(queueName, false, consumer);
By changing to the "pull" method, Priority seems to be respected. However, in the quote below (from the same url above), it looks like there is a trade-off (that I've bolded)
Fetching Individual Messages ("pull API")
To retrieve individual messages, use IModel.BasicGet. The returned value is an instance of BasicGetResult, from which the header information (properties) and message body can be extracted:
bool noAck = false;
BasicGetResult result = channel.BasicGet(queueName, noAck);
if (result == null) {
// No message available at this time.
} else {
IBasicProperties props = result.BasicProperties;
byte[] body = result.Body;
...
Since noAck = false above, you must also call IModel.BasicAck to acknowledge that you have successfully received and processed the message:
...
// acknowledge receipt of the message
channel.BasicAck(result.DeliveryTag, false);
}
Note that fetching messages using this API is relatively inefficient. If you'd prefer RabbitMQ to push messages to the client, see the next section.
(The "next" section in this case takes you to the "push" method at the top of this post)
Related
The Erlang RMQ server runs non-stop.
Let's say I have a RMQ receiver and RMQ sender which sends a message right from the start.
When I start them: receiver first, sender second, the message is sent and server picks it up.
But when I start sender first, and right after it sends its message, I start receiver, receiver does not see that message.
My question is is RMQ capable of handling (reading) the message in the second case, and if yes what are the required options? The exchange is durable already but it didn't help.
YES, RabbitMQ is capable of keeping those messages until your consumer becomes available. It is just a matter of when have you declared the queue, and the queue configuration.
In short; If you declare and bind your queue before the messages are published, the messages will be kept in that queue regardless of not having a consumer yet.
As explained in the documentation, some parameters affect this behavior when declaring a queue:
exclusive, if true, will delete the queue (and the messages) when the consumer disconnects.
durable, if true, will keep the messages even when RabbitMQ restarts.
When the consumers create and setup their own queues idempotently on startup, the situation is exactly like you describe; Messages get "lost" if the consumer has never ran.
This is somewhat a common practice as you can verify by the tutorials, especially in a pub-sub scenario (see tutorial #3) where publishers don't know about the consumers or what queues they would need.
If you really want to make sure all messages are received, some means of declaring the queues beforehand is needed.
The simple producer
// producer
static void Main( string[] args )
{
var factory = new ConnectionFactory()
{
HostName = "localhost"
};
using ( var connection = factory.CreateConnection() )
{
using ( var channel = connection.CreateModel() )
{
channel.QueueDeclare( "hello", true, false, false, null );
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes( message );
while ( true )
{
Console.WriteLine( "Press key to send message" );
Console.ReadLine();
channel.BasicPublish( "", "hello", null, body );
Console.WriteLine( " [x] Sent {0}", message );
}
}
}
I run this, publish few messages and I see them all in a queue at the RMQ.
Then I run this simple consumer
// consumer
static void Main( string[] args )
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", true, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
}
}
and I got my messages in the consumer even though it's been run after the producer finished.
Please verify how does this differ from your setup.
Edit: This also works if there's an exchange backing the queue in the producer:
// producer
static void Main( string[] args )
{
var factory = new ConnectionFactory()
{
HostName = "localhost"
};
using ( var connection = factory.CreateConnection() )
{
using ( var channel = connection.CreateModel() )
{
channel.QueueDeclare( "hello", true, false, false, null );
channel.ExchangeDeclare( "helloe", "fanout", true );
channel.QueueBind( "hello", "helloe", "" );
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes( message );
while ( true )
{
Console.WriteLine( "Press key to send message" );
Console.ReadLine();
channel.BasicPublish( "helloe", "", null, body );
Console.WriteLine( " [x] Sent {0}", message );
}
}
}
Console.WriteLine( "finished" );
Console.ReadLine();
}
(no changes to the consumer are necessary)
Edit 2: this also works when eventing consumer is used:
static void Main( string[] args )
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", true, false, false, null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (s, e) =>
{
var message = Encoding.UTF8.GetString(e.Body);
Console.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
Console.ReadLine();
}
}
}
I've been looking through several examples of the Confluent.Kafka client (https://github.com/confluentinc/confluent-kafka-dotnet/), and whilst I can successfully get a producer to push a message into Kafka, I'm unable to pull any messages back down with consumers.
Through the UI I can see the topic is created and messages are going into this topic (there are currently 10 partitions and 3 messages), but my consumer always reports "end of partition", without any consumption of any message (the 3 remain on the topic and "OnMessage" never fires).
However the consumer is definitely accessing the topics, and can see 3 messages on one of the partitions:
end of partition: dotnet-test-topic [6] #3
It just doesn't consume the message and trigger OnMessage(). Any ideas?
var conf = new Dictionary<string, object>
{
{ "group.id", Guid.NewGuid().ToString() },
{ "bootstrap.servers", "mykafkacluster:9094" },
{ "sasl.mechanisms", "SCRAM-SHA-256" },
{ "security.protocol", "SASL_SSL" },
{ "sasl.username", "myuser" },
{ "sasl.password", "mypass" }
};
using (var producer = new Producer<string, string>(conf, new StringSerializer(Encoding.UTF8), new StringSerializer(Encoding.UTF8)))
{
producer.ProduceAsync("dotnet-test-topic", "some key", "some value")
.ContinueWith(result =>
{
var msg = result.Result;
if (msg.Error.Code != ErrorCode.NoError)
{
Console.WriteLine($"failed to deliver message: {msg.Error.Reason}");
}
else
{
Console.WriteLine($"delivered to: {result.Result.TopicPartitionOffset}");
}
});
producer.Flush(TimeSpan.FromSeconds(10));
}
using (var consumer = new Consumer<string, string>(conf, new StringDeserializer(Encoding.UTF8), new StringDeserializer(Encoding.UTF8)))
{
consumer.Subscribe("dotnet-test-topic");
consumer.OnConsumeError += (_, err)
=> Console.WriteLine($"consume error: {err.Error.Reason}");
consumer.OnMessage += (_, msg)
=> Console.WriteLine($"consumed: {msg.Value}");
consumer.OnPartitionEOF += (_, tpo)
=> Console.WriteLine($"end of partition: {tpo}");
while (true)
{
consumer.Poll(TimeSpan.FromMilliseconds(100));
}
}
Looks like the OnMessage event won't fire without the following config provided:
{ "auto.offset.reset", "smallest" }
With this added, I was able to read the messages on the topic.
I was able to consume multiple messages that are sent by multiple producers to the same exchange with different routing key using the above code and was able to insert each message to database.
But this will consume too much of resources as messages will be inserted into DB one after the other. So I decided to go for batch insert and I found I can set BasicQos
After setting the message limit to 10 in BasicQos, my expectation is the Console.WriteLine must write 10 messages, but it is not as expected.
My expectation is to consume N number messages from the queue and do bulk insert and on successful send ACK else No ACK
Here is the piece of code I use.
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_A");
channel.QueueBind(queue: "queueName", exchange: "exchangeName", routingKey: "Producer_B");
channel.BasicQos(0, 10, false);
var consumer = new EventingBasicConsumer(channel);
channel.BasicConsume(queue: "queueName", noAck: false, consumer: consumer);
consumer.Received += (model, ea) =>
{
try
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
// Insert into Database
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
Console.WriteLine(" Recevier Ack " + ea.DeliveryTag);
}
catch (Exception e)
{
channel.BasicNack(deliveryTag: ea.DeliveryTag, multiple: false, requeue: true);
Console.WriteLine(" Recevier No Ack " + ea.DeliveryTag);
}
};
Console.ReadLine();
}
}
BasicQos = 10 means that the client fetch only 10 messages at time, but when you consume it you will see always one message a time.
Read here: https://www.rabbitmq.com/consumer-prefetch.html
AMQP specifies the basic.qos method to allow you to limit the number
of unacknowledged messages on a channel (or connection) when consuming
(aka "prefetch count").
for your scope you have to download the messages, put it inside a temporary list and then insert into the DB.
an then you can use:
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: true);
void basicAck()
Parameters:
deliveryTag - the tag from the received
AMQP.Basic.GetOk or AMQP.Basic.Deliver
multiple - true to acknowledge
all messages up to and including the supplied delivery tag; false to
acknowledge just the supplied delivery tag.
Example
final List<String> myMessagges = new ArrayList<String>();
channel.basicConsume("my_queue", false, new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
myMessagges.add(new String(body));
System.out.println("Received...");
if (myMessagges.size() >= 10) {
System.out.println("insert into DB...");
channel.basicAck(envelope.getDeliveryTag(), true);
myMessagges.clear();
}
}
});
Batch size based consumption can be done using the channel.basicQos().
Channel channel = connection.createChannel();
channel.basicQos(10);
It specifies the maximum no of messages to be fetched without sending ACK for each.
Use the DefaultConsumer class and override its methods.
Consumer batchConsumer = new DefaultConsumer(channel) {
#Override
public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
}
#Override
public void handleCancelOk(String consumerTag) {
}
};
Consume 10 messages using channel.basicConsume()
channel.basicConsume(QUEUE_NAME, false, batchConsumer);
When channel.basicConsume() is called it will fetch a batch of 10 messages. 'false' is set to disable auto ack, and ACK to be sent only once after consuming entire batch.
channel.basicAck(getLastMessageEnvelope().getDeliveryTag(), true);
Here 'true' means we are sending ACK for multiple messages.
Detailed explanation can be found in
RabbitMQ Batch Consumption
MessageType: "PublishX"
Consumers:
Type1ConsumerX
Type2ConsumerX
Type3ConsumerX
All of the consumers must catch messages immediately, but consume synchronously inside themselves..
For example there are 100 "PublishX" messages in the queue. Type1ConsumerX consumed 30 messages (synchronously), Type2ConsumerX consumed 50 messages(synchronously) , Type3ConsumerX consumed 100 messages(synchronously).
How can I know the message is consumed by "all type of consumers" ?
Could RabbitMQ/MassTransit PUSH messages to consumers?
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
If I've understood the question correctly you just need to set up a basic pub/sub pattern. This will allow you to deliver the same message to multiple consumers.
Example publisher:
public static void PublishMessageToFanout()
{
var factory = new ConnectionFactory { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
var message = new Message { Text = "This is a message to send" };
var json = JsonConvert.SerializeObject(message);
var body = Encoding.UTF8.GetBytes(json);
channel.BasicPublish("messages", string.Empty, null, body);
}
}
Example consumers:
SubscribeToMessages("sms-messages", (s) => Console.WriteLine("SMS Message: {0}", s));
SubscribeToMessages("email-messages", (s) => Console.WriteLine("Email Message: {0}", s));
public static void SubscribeToMessages(string queueName, Action<string> messageAction)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
channel.ExchangeDeclare("messages", "fanout");
channel.QueueDeclare(queueName, true, false, false, null);
channel.QueueBind(queueName, "messages", string.Empty);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(queueName, true, consumer);
while (true)
{
var ea = consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
messageAction(message);
}
}
}
If you run the SubscribeToMessages loops in separate processes or console apps you'll see that they both print out the message whenever you call the PublishMessageToFanout. You'll also see that both queues exist in RabbitMQ Management under Queues.
Regarding the MassTransit part of your question
RabbitMQ/MassTransit PUSH messages to consumers?
Yes, MassTransit publishes messages to the bus, and then a consumer processes them
Could RabbitMQ/MassTransit push messages (merging them) with intervals (1s) for decrease network traffic?
Don't know if there is a feature for this, you could write your own but you would have to be very careful about losing the messages.
Could RabbitMQ/MassTransit push same messages to the different type of Consumers?
Yes, multiple consumers can consume the same type of message.
I've written a simple hello world app that shows the basics - http://nodogmablog.bryanhogan.net/2015/04/mass-transit-with-rabbitmq-hello-world/
The basic RabbitMQ tutorial gives an example of how to retrieve messages continuously from a queue:
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", false, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
}
What I want to do is retrieve all messages which have been placed onto the queue and then stop.
Here are two examples that would solve my problem
If I start my code at 1pm, I want to process all the messages that have been placed on the queue before 1pm.
OR
If I start my code at 13:00:00, and it takes 10 seconds for my code to run, I don't mind if it includes messages placed on the queue between 13:00:00 and 13:00:10, as long as it stops as soon as the queue is empty.
I realize that I can probably put a time stamp in my message and check for that, or I could fiddle with timeout values, but I was wondering if there's any built in way to do this properly.
Thanks in advance.
From the comments it seems that RabbitMQ is not meant for batch processing, so it hasn't been designed for this purpose.
I've also noticed that, when testing the DequeueNoWait method, or attempting to Dequeue with a zero timeout, they didn't work at all, and simply returned null.
The following solution uses QueueDeclare to get a count of existing messages and doesn't require a time stamp or a hacky timeout:
var factory = new ConnectionFactory { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
var queueDeclareResponse = channel.QueueDeclare(Constants.QueueName, false, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume(Constants.QueueName, true, consumer);
Console.WriteLine(" [*] Processing existing messages.");
for (int i = 0; i < queueDeclareResponse.MessageCount; i++)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
Console.WriteLine("Finished processing {0} messages.", queueDeclareResponse.MessageCount);
Console.ReadLine();
}
}
What you can do:
1.) First add a timestamp to the message object
2.) Reject the message if the timestamp is not valid. It is an integrated feature of rabbitmq
Reference