Develop Reference

Reading message sent before start?

The Erlang RMQ server runs non-stop.
Let's say I have a RMQ receiver and RMQ sender which sends a message right from the start.
When I start them: receiver first, sender second, the message is sent and server picks it up.
But when I start sender first, and right after it sends its message, I start receiver, receiver does not see that message.
My question is is RMQ capable of handling (reading) the message in the second case, and if yes what are the required options? The exchange is durable already but it didn't help.

YES, RabbitMQ is capable of keeping those messages until your consumer becomes available. It is just a matter of when have you declared the queue, and the queue configuration.
In short; If you declare and bind your queue before the messages are published, the messages will be kept in that queue regardless of not having a consumer yet.
As explained in the documentation, some parameters affect this behavior when declaring a queue:
exclusive, if true, will delete the queue (and the messages) when the consumer disconnects.
durable, if true, will keep the messages even when RabbitMQ restarts.
When the consumers create and setup their own queues idempotently on startup, the situation is exactly like you describe; Messages get "lost" if the consumer has never ran.
This is somewhat a common practice as you can verify by the tutorials, especially in a pub-sub scenario (see tutorial #3) where publishers don't know about the consumers or what queues they would need.
If you really want to make sure all messages are received, some means of declaring the queues beforehand is needed.

The simple producer
// producer
static void Main( string[] args )
{
var factory = new ConnectionFactory()
{
HostName = "localhost"
};
using ( var connection = factory.CreateConnection() )
{
using ( var channel = connection.CreateModel() )
{
channel.QueueDeclare( "hello", true, false, false, null );
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes( message );
while ( true )
{
Console.WriteLine( "Press key to send message" );
Console.ReadLine();
channel.BasicPublish( "", "hello", null, body );
Console.WriteLine( " [x] Sent {0}", message );
}
}
}
I run this, publish few messages and I see them all in a queue at the RMQ.
Then I run this simple consumer
// consumer
static void Main( string[] args )
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", true, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
}
}
}
}
and I got my messages in the consumer even though it's been run after the producer finished.
Please verify how does this differ from your setup.
Edit: This also works if there's an exchange backing the queue in the producer:
// producer
static void Main( string[] args )
{
var factory = new ConnectionFactory()
{
HostName = "localhost"
};
using ( var connection = factory.CreateConnection() )
{
using ( var channel = connection.CreateModel() )
{
channel.QueueDeclare( "hello", true, false, false, null );
channel.ExchangeDeclare( "helloe", "fanout", true );
channel.QueueBind( "hello", "helloe", "" );
string message = "Hello World!";
var body = Encoding.UTF8.GetBytes( message );
while ( true )
{
Console.WriteLine( "Press key to send message" );
Console.ReadLine();
channel.BasicPublish( "helloe", "", null, body );
Console.WriteLine( " [x] Sent {0}", message );
}
}
}
Console.WriteLine( "finished" );
Console.ReadLine();
}
(no changes to the consumer are necessary)
Edit 2: this also works when eventing consumer is used:
static void Main( string[] args )
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("hello", true, false, false, null);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (s, e) =>
{
var message = Encoding.UTF8.GetString(e.Body);
Console.WriteLine(" [x] Received {0}", message);
};
channel.BasicConsume("hello", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
Console.ReadLine();
}
}
}

When Kafka Consumer subscribe the event, it show Timed out 1 in-flight

This script is the method to subscribe event from Kafka.
using Confluent.Kafka;
using Confluent.Kafka.Serialization;
static void Main(string[] args)
{
string brokerList = "broker";
var topics = new List<string>() { "topicName" };
var config = new Dictionary<string, object>
{
{ "group.id", "ConsumerGroup" },
{ "bootstrap.servers", brokerList },
{ "auto.offset.reset", "earliest" },
{ "enable.auto.commit", false }
};
using (var consumer = new Consumer<string, string>(config, new StringDeserializer(Encoding.UTF8), new StringDeserializer(Encoding.UTF8)))
{
consumer.OnMessage += (obj, msg) =>
{
...
};
consumer.Subscribe(topics);
while (true)
{
consumer.Poll(TimeSpan.FromMilliseconds(1000));
}
}
}
When I trace the code in Debug mode, the order of subscribing event is:
consumer.Subscribe(topics)
consumer.Poll(TimeSpan.FromMilliseconds(1000));
consumer.OnMessage += (obj, msg) =>
Before getting the new event (go to consumer.OnMessage), it spent a little time to poll (in consumer.Poll) and print some information on console window.
As follow:
4|2018-12-12 10:41:53.381|rdkafka#consumer-1|REQTMOUT| [thrd:broker/bootstrap]: broker/bootstrap: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
In my original thoughts, it use consumer.Subscribe(topics) to connect broker and use consumer.Poll to consume the new event.
But it seems that the consumer.Poll includes connecting to broker and consuming the new event.
My questions are:
Which function can connect to broker? consumer.Subscribe or
consumer.Poll or?
Why consumer.Poll print the information on console window? And it seems that having some error (Timed out 1 in-flight).

About your first question.
Which function can connect to broker? consumer.Subscribe or consumer.Poll or?
consumer.Subscribe connect to broker and consumer.Poll consume messages.
And about the second one.
Why consumer.Poll print the information on console window? And it seems that having some error (Timed out 1 in-flight).
Confluent.Kafka has released in a new major version in here. If you have any problem with previous versions you can use the recommended version. based on the GitHub repository:
It has more features, is considerably improved and is more performant than 0.11.x releases

how to respond to an event on a different thread

I have implemented a rabbitMQ listener, which essentially just sits in a loop waiting for messages to arrive. When the message arrives I wish to fire an event and run some code.
However I don't always seem to receive this event, and I am wondering if this is because I am running the queue polling part of the code on a different thread.
It does seem to work initially though, so it is possible that the threading is not the problem. Can anyone give me an opinion on this?
QueueListener:
public void CreateQueueListener<T>() where T : IPubSubEvent
{
var mqServer = new RabbitMqServer(m_RabbitMqAddress);
var mqClient = (RabbitMqQueueClient)mqServer.MessageFactory.CreateMessageQueueClient();
var channel = mqClient.Channel;
string queueName = mqClient.GetTempQueueName();
channel.QueueBind(queueName, m_EventExchange, routingKey: QueueNames<T>.In);
var consumer = new RabbitMqBasicConsumer(channel);
channel.BasicConsume(queue: queueName, autoAck: true, consumer: consumer);
Task.Run(() =>
{
while (true)
{
BasicGetResult basicGetResult;
try
{
basicGetResult = consumer.Queue.Dequeue();
}
catch (Exception ex)
{
throw;
}
var message = basicGetResult.ToMessage<T>();
PublishEvent?.Invoke(this, new PubSubEventArgs { EventData = message.GetBody().EventName });
}
});
}
Consuming Class
public class MyClass
{
public MyClass(IEventClient eventClient)
{
eventClient.CreateQueueListener<AuthoriseEvent>();
eventClient.PublishEvent += OnPublishEvent;
}
private async void OnPublishEvent(object sender, PubSubEventArgs e)
{
if (e.EventData == "AuthoriseEvent")
//dostuff
}
}

I am running the queue polling part of the code on a different thread
As far as I know, this isn't supported by the .NET client.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

RabbitMQ only listens to the first message on a queue

I'm having an issue with my Rabbit queues that is currently only reacting to the first message in queue, after that any other messages being pushed are being ignored.
I start with instantiating the connection and declaring the queue in my IQueueConnectionProvider:
var connectionFactory = new ConnectionFactory() { HostName = hostName };
var connection = _connectionFactory.CreateConnection();
var channel = connection.CreateModel();
That IQueueConnectionProvider is then used in my IQueueListener as a dependency with just one method:
public void ListenToQueue(string queue)
{
var channel = _queueConnectionProvider.GetQueue();
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
string path = #"d:\debug.log.txt";
File.AppendAllLines(path, new List<string>() {"MESSAGE RECEIVED", Environment.NewLine });
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
channel.BasicAck(ea.DeliveryTag, false);
};
channel.BasicConsume(queue, true, consumer);
}
My log file ends up being just one line "MESSAGE RECEIVED", however I can see in the Rabbit ui interface that my other services are pushing the messages to that queue just fine.
Is there something I'm missing here?

This was a dumb mistake... yet again.
channel.BasicConsume(queue, false, consumer);
This was what I needed. I want to manually acknowledge my messages, therefore noAck needs to be false;

The code works fine! Have tested with my queue, and was able to get "MESSAGE RECEIVED" 9 times in the log file; since I had 9 messages in my queue.
I tried without this line of code, and it worked fine
var channel = _queueConnectionProvider.GetQueue();

RabbitMQ 3.5 and Message Priority

RabbitMQ 3.5 now supports message priority;
However, I am unable to build a working example. I've placed my code below. It includes the output that I expect and the output I actually. I'd be interested in more documentation, and/or a working example.
So my question in short: How do I get message priority to work in Rabbit 3.5.0.0?
Publisher:
using System;
using RabbitMQ.Client;
using System.Text;
using System.Collections.Generic;
class Publisher
{
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
IDictionary <String , Object> args = new Dictionary<String,Object>() ;
args.Add(" x-max-priority ", 10);
channel.QueueDeclare("task_queue1", true, false, true, args);
for (int i = 1 ; i<=10; i++ )
{
var message = "Message";
var body = Encoding.UTF8.GetBytes(message + " " + i);
var properties = channel.CreateBasicProperties();
properties.SetPersistent(true);
properties.Priority = Convert.ToByte(i);
channel.BasicPublish("", "task_queue1", properties, body);
}
}
}
}
}
Consumer:
using System;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using System.Text;
using System.Threading;
using System.Collections.Generic;
namespace Consumer
{
class Worker
{
public static void Main()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
IDictionary<String, Object> args = new Dictionary<String, Object>();
channel.BasicQos(0, 1, false);
var consumer = new QueueingBasicConsumer(channel);
IDictionary<string, object> consumerArgs = new Dictionary<string, object>();
channel.BasicConsume( "task_queue1", false, "", args, consumer);
Console.WriteLine(" [*] Waiting for messages. " +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
channel.BasicAck(ea.DeliveryTag, false);
}
}
}
}
}
}
Actual output:
[*] Waiting for messages. To exit press CTRL+C
[x] Received Message 1
[x] Received Message 2
[x] Received Message 3
[x] Received Message 4
[x] Received Message 5
[x] Received Message 6
[x] Received Message 7
[x] Received Message 8
[x] Received Message 9
[x] Received Message 10
Expected output:
[*] Waiting for messages. To exit press CTRL+C
[x] Received Message 10
[x] Received Message 9
[x] Received Message 8
[x] Received Message 7
[x] Received Message 6
[x] Received Message 5
[x] Received Message 4
[x] Received Message 3
[x] Received Message 2
[x] Received Message 1
UPDATE #1.
I found an example in Java here. However it's the Rabbit 3.4.x.x. addin that was incorporated into 3.5.
The only difference I can see is that they express the priority as an int and mine is a byte. But I feel like that's a red herring. I'm at a bit of a loss here.

Well I solved it.
It was a dumb mistake.
I wrote:
args.Add(" x-max-priority ", 10);
It should have been
args.Add("x-max-priority", 10);
I'll leave this up so other people can have a working example of Rabbitmq 3.5's Priority Queues in C#.

A similar RabbitMq Priority Queue Implementation in Node JS
Install amqplib
In order to test, we are required to have amqplib installed
npm install amqplib
Publisher (send.js)
#!/usr/bin/env node
var amqp = require('amqplib/callback_api');
function bail(err, conn) {
console.error(err);
if (conn) conn.close(function() { process.exit(1); });
}
function on_connect(err, conn) {
if (err !== null) return bail(err);
// name of queue
var q = 'hello';
var msg = 'Hello World!';
var priorityValue = 0;
function on_channel_open(err, ch) {
if (err !== null) return bail(err, conn);
// maxPriority : max priority value supported by queue
ch.assertQueue(q, {durable: false, maxPriority: 10}, function(err, ok) {
if (err !== null) return bail(err, conn);
for(var index=1; index<=100; index++) {
priorityValue = Math.floor((Math.random() * 10));
msg = 'Hello World!' + ' ' + index + ' ' + priorityValue;
ch.publish('', q, new Buffer(msg), {priority: priorityValue});
console.log(" [x] Sent '%s'", msg);
}
ch.close(function() { conn.close(); });
});
}
conn.createChannel(on_channel_open);
}
amqp.connect(on_connect);
Subscriber (receive.js)
#!/usr/bin/env node
var amqp = require('amqplib/callback_api');
function bail(err, conn) {
console.error(err);
if (conn) conn.close(function() { process.exit(1); });
}
function on_connect(err, conn) {
if (err !== null) return bail(err);
process.once('SIGINT', function() { conn.close(); });
var q = 'hello';
function on_channel_open(err, ch) {
ch.assertQueue(q, {durable: false, maxPriority: 10}, function(err, ok) {
if (err !== null) return bail(err, conn);
ch.consume(q, function(msg) { // message callback
console.log(" [x] Received '%s'", msg.content.toString());
}, {noAck: true}, function(_consumeOk) { // consume callback
console.log(' [*] Waiting for messages. To exit press CTRL+C');
});
});
}
conn.createChannel(on_channel_open);
}
amqp.connect(on_connect);
Run:
node send.js
It will create a queue named 'hello' and will flood it with '1000' sample messages using default AMQP exchange.
node receive.js
It will act as a consumer to subscribe to messages waiting in the queue.

Another possibility (for future searchers)
The "Push" method of message delivery doesn't seem to respect Priority.
http://rabbitmq.docs.pivotal.io/35/rabbit-web-docs/dotnet-api-guide.html.html
The below is a quote from the URL above. I've bolded the important part.
Retrieving Messages By Subscription ("push API")
Another way to receive messages is to set up a subscription using the IBasicConsumer interface. The messages will then be delivered automatically as they arrive, rather than having to be requested proactively. One way to implement a consumer is to use the convenience class EventingBasicConsumer, which dispatches deliveries and other consumer lifecycle events as C# events:
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (ch, ea) =>
{
var body = ea.Body;
// ... process the message
ch.BasicAck(ea.DeliveryTag, false);
};
String consumerTag = channel.BasicConsume(queueName, false, consumer);
By changing to the "pull" method, Priority seems to be respected. However, in the quote below (from the same url above), it looks like there is a trade-off (that I've bolded)
Fetching Individual Messages ("pull API")
To retrieve individual messages, use IModel.BasicGet. The returned value is an instance of BasicGetResult, from which the header information (properties) and message body can be extracted:
bool noAck = false;
BasicGetResult result = channel.BasicGet(queueName, noAck);
if (result == null) {
// No message available at this time.
} else {
IBasicProperties props = result.BasicProperties;
byte[] body = result.Body;
...
Since noAck = false above, you must also call IModel.BasicAck to acknowledge that you have successfully received and processed the message:
...
// acknowledge receipt of the message
channel.BasicAck(result.DeliveryTag, false);
}
Note that fetching messages using this API is relatively inefficient. If you'd prefer RabbitMQ to push messages to the client, see the next section.
(The "next" section in this case takes you to the "push" method at the top of this post)