Problem with RabbitMQ and deduplication plugin - c#

I'm a RabbitMQ newbie; for a new project I need to use the deduplication plugin. I'm using AspNet Core 3.0 worker process and language is C#.
I've tried a very simple example, 2 publishers sending 10 messages numbered 1 to 10 and one consumer getting messages and acknowledging them.
I'm having quite strange and unpredictable results:
if I run the 3 workers (2 Publishers and one consumer) inside the same process, it looks like that deduplication plugin works fine and inserts in the queue only 10 unique messages, but the consumer reads only the first 2 and ackowledges only one of them.
if I run publishers and consumer in two different processes, the consumer gets all the 10 messages but after ack the messages remain in the queue and if I run again the consumer process they get reprocessed again.
I've tried to google for some full working sample in C# for deduplication, but without success
Publisher
int cnt = 1;
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
Dictionary<string, object> dd = new Dictionary<string, object>();
dd["x-message-deduplication"] = true;
channel.QueueDeclare(queue: qname,
durable: true,
exclusive: false,
autoDelete: false,
arguments: dd);
while (!stoppingToken.IsCancellationRequested)
{
var message = GetMessage(cnt);
var body = Encoding.UTF8.GetBytes(message);
var properties = channel.CreateBasicProperties();
properties.Persistent = true;
Dictionary<string, object> d = new Dictionary<string, object>();
d["x-deduplication-header"] = cnt;
properties.Headers = d;
channel.BasicPublish(exchange: "",
routingKey: qname,
basicProperties: properties,
body: body);
Console.WriteLine(" [x] Sent {0}", message);
logDB(cnt, "Sender"+Wname);
cnt++;
if (cnt > 10)
break;
await Task.Delay(1000, stoppingToken);
}
Consumer:
while (!stoppingToken.IsCancellationRequested)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
{
Dictionary<string, object> dd = new Dictionary<string, object>();
dd["x-message-deduplication"] = true;
channel.QueueDeclare(queue: qname,
durable: true,
exclusive: false,
autoDelete: false,
arguments: dd);
_logger.LogInformation("{0} Waiting for messages.", Cname);
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
_logger.LogInformation("{0} Received {1}", Cname, message);
string[] parts = message.Split('-');
int cntmsg = int.Parse(parts[1]);
logDB(cntmsg, Cname);
Thread.Sleep((cntmsg % 5) * 1000);
_logger.LogInformation("{0} Received {1} done", Cname, message);
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: true);
};
channel.BasicConsume(queue: qname,
autoAck: false,
consumer: consumer);
_logger.LogInformation("{0} After BasicConsume", Cname);
while (true)
await Task.Delay(1000, stoppingToken);
}

after contacting the developer of the deduplication plugin it turned out the problem was related to the type (int) of deduplication header, using a string value works.
He will release a new version supporting int datatype soon.

Related

Queue is locking even though it is declared as non-exclusive (RabbitMQ)

I'm using the .NET client for RabbitMQ to implement the RPC pattern for a client and server. The client and server both declare the queues using the same parameters so that messages sent before the server is up won't be lost and vice versa. However, declaring the queue on the client side throws an OperationInterruptedException:
operation queue.declare caused a channel exception resource_locked: cannot obtain exclusive access to locked queue '151' in vhost '/'. It could be originally declared on another connection or the exclusive property value does not match that of the original declaration.
This question is similar to RabbitMQ: ACCESS_REFUSED even if the queue is non-exclusive, but the answer there did not work for me. The BasicConsumer defaults to being non-exclusive in the .NET client. If I declare the queue only in the server, the code works without issues. I don't want to have to make sure the server starts before the client, though.
What is causing this exception? Neither queue is declared as exclusive.
(Both code samples have been compressed for brevity.)
Client:
public MyClient(){
try {
ConnectionFactory factory = new() {
HostName = "localhost"
};
IConnection connection = factory.CreateConnection();
channel = connection.CreateModel();
sendUpdatesMessenger = new("sendUpdates", channel);
getUpdatesMessenger = new("getUpdates", channel);
getUpdatesReceiver = new("getUpdates", "localhost", channel);
sendUpdatesReceiver = new("sendUpdates", "localhost", channel);
string bodyStr = "test example string";
string messageId = new Guid().ToString();
bool success = getUpdatesMessenger.SendMessage(messageBody: bodyStr, routingKey: "151", correlationId: messageId, replyQueue: "myReplyQueue");
//Do stuff based on success or failure
} catch (Exception ex){
Debug.WriteLine("Exception " + ex);
}
}
public bool SendMessage(string messageBody, string routingKey, string correlationId, string replyQueue = ""){
try{
IBasicProperties props = channel.CreateBasicProperties();
props.CorrelationId = correlationId;
props.ReplyTo = replyQueue;
channel.QueueDeclare(queue: routingKey, durable: true, exclusive: false, autoDelete: false);
channel.BasicPublish(exchange: exchange, routingKey: routingKey, basicProperties: props, body: Encoding.ASCII.GetBytes(messageBody));
return true;
}
catch(Exception ex){
Debug.WriteLine("Error: " + ex);
return false;
}
}
Server:
public bool BindQueue(string routingKey){
try {
channel.QueueDeclare(queue: routingKey, durable: true, autoDelete: false);
channel.QueueBind(queue: routingKey, routingKey: routingKey, exchange: exchangeName);
queueNames.Add(routingKey);
return true;
}
catch (Exception ex){
Console.WriteLine(ex);
return false;
}
}

How to send acknowledgment (Consumer) in RabbitMQ externally?

I have an application where it sending message to RMQ broker as below:
var connectionFactory = new ConnectionFactory()
{
HostName = "localhost"
};
using (var connection = connectionFactory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("demo",
durable:true,
exclusive:false,
autoDelete:false,
arguments:null);
Console.WriteLine("Click enters to send random case Id");
do
{
Console.ReadLine();
var message = new {CaseId = new Random().Next()};
var body = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(message));
channel.BasicPublish("", "demo", null, body);
Console.WriteLine("Successfully send message.");
} while (true);
}
}
It successfully sent the message.
There is another app called the consumer app.
The code is below:
private void InitiateRabbitMq()
{
var connectionFactory = new ConnectionFactory()
{
HostName = "localhost"
};
var connection = connectionFactory.CreateConnection();
var channel = connection.CreateModel();
MessageHandler messageReceiver = new MessageHandler(channel);
channel.BasicConsume("demo", false, messageReceiver);
}
The message handler is:
public class MessageHandler : DefaultBasicConsumer
{
private readonly IModel _channel;
public MessageHandler(IModel channel)
{
_channel = channel;
}
public override async void HandleBasicDeliver(string consumerTag, ulong deliveryTag, bool redelivered, string exchange, string routingKey,
IBasicProperties properties, ReadOnlyMemory<byte> body)
{
var message = JsonConvert.DeserializeObject(Encoding.UTF8.GetString(body.ToArray()));
_processor.process(message);
}
}
Here is the process method parth which is another class:
client.BeginTransfer(transfer,
HandleUploadProgressChanged,
HandleUploadComplete,
HandleUploadInterrupted,
HandleUploadCancelled,
3600,
UploadFilesFinishedCallback);
Once begin transfer completed. It invoked UploadFilesFinishedCallback . I want to acknowledge in this method. How can I do it?
This is how you Acknowledge message:
channel.BasicAck(deliveryTag, false);
So it means your Function "UploadFilesFinishedCallback" must have the deliveryTag
==> and this means that also your "process" function must have the deliveryTag (which currently gets only the message content)
solution:
add new parameter "deliveryTag" to function "process", and to function "UploadFilesFinishedCallback"
you can use it in the callback like this:
client.BeginTransfer(transfer,
HandleUploadProgressChanged,
HandleUploadComplete,
HandleUploadInterrupted,
HandleUploadCancelled,
3600,
() => { UploadFilesFinishedCallback(deliveryTag) });
(depends on the signature of the callback function)

C# Ping is not fast enough

I am trying to make a scan wether the pc is online or offline.
But my current code is way to slow to scan with a good performance as if an computer is offline there is a delay of 3 to 5 seconds.
I even added the timeout parameter set as 500 but it still takes more than 3 seconds if a computer is offline.
public bool PingComputer(string computername)
{
bool check = false;
Ping ping = new Ping();
try
{
PingReply reply = ping.Send(computername, 500);
check = reply.Status == IPStatus.Success;
}
catch (PingException)
{
}
return check;
}
I also already read about asynchron pings but i couldĀ“nt find a suitable solution yet that simply returns true if computer is online or false if it is offline.
Thanks in advance.
If you pretended to Ping a list of Computers you can use Parallel or use async Task.
I tested both methods bellow with the same 77 IPs. Used variable sec = 3.
Tasks toke 00:00:02.7146249
Parallel toke 00:00:05.9941404
To use the methods
Dictionary<string, bool> pingsReturn = await Network.PingListAsync(dictionary,3);
I can give you the 2 examples:
Task
public static async Task<Dictionary<string, bool>> PingListAsync(Dictionary<string, bool> HostList, int sec = 3)
{
// Create a buffer of 32 bytes of data to be transmitted.
string data = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
byte[] buffer = Encoding.ASCII.GetBytes(data);
// set a quick TTL
PingOptions options = new PingOptions(20, true);
// internal support Task to handle Ping Exceptions like "host not found"
async Task<KeyValuePair<string, bool>> PingHost(string host)
{
try
{
var pingresult = await Task.Run(() => new Ping().SendPingAsync(host, sec * 1000, buffer, options));
//t.Wait();
if (pingresult.Status == IPStatus.Success)
return new KeyValuePair<string, bool>(host, true);
else
return new KeyValuePair<string, bool>(host, false);
}
catch
{
return new KeyValuePair<string, bool>(host, false);
}
}
//Using Tasks >>
var watch = new Stopwatch();
watch.Start();
var tasksb = HostList.Select(HostName => PingHost(HostName.Key.ToString()));
var pinglist = await Task.WhenAll(tasksb);
foreach (var pingreply in pinglist)
{
HostList[pingreply.Key] = pingreply.Value;
}
watch.Stop();
Log.Debug("PingList (Tasks) Time elapsed: " + watch.Elapsed);
//Using Tasks <<
return HostList;
}
Parallel
public static async Task<Dictionary<string, bool>> PingListAsync(Dictionary<string, bool> HostList, int sec = 3)
{
// Create a buffer of 32 bytes of data to be transmitted.
string data = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
byte[] buffer = Encoding.ASCII.GetBytes(data);
// set a quick TTL
PingOptions options = new PingOptions(20, true);
//Using Parallel >>
watch = new Stopwatch();
watch.Start();
// avoid exception "Collection was modified; enumeration operation may not execute."
// we need new dictionary and add values
Dictionary<string, bool> dictionary = new Dictionary<string, bool>();
Parallel.ForEach(HostList.Keys, (currHost) =>
{
try
{
var pingReply = new Ping().Send(currHost, sec * 1000, buffer, options);
if (pingReply.Status == IPStatus.Success)
dictionary.Add(currHost, true);
else
dictionary.Add(currHost, false);
}
catch
{
// handle Ping Exceptions like "host not found"
dictionary.Add(currHost, false);
}
});
watch.Stop();
Log.Debug("PingList (Parallel) Time elapsed: " + watch.Elapsed);
//Using Parallel <<
return dictionary;
}
PS - I know this is an old question, but is still valid.

Wait until a message has been received and pass the message body using C# RabbitMQ

I want to read a message from a queue, and once available wants to send the byte[] outside the consumer class.
public byte[] Receive()
{
if (messagingAdapter == null)
return default(byte[]);
byte[] messageBody = null;
var channel = messagingAdapter.GetChannel();
channel.BasicQos(prefetchSize: 0, prefetchCount: 1, global: false);
using (var subscription = new Subscription(channel, containerName, false))
{
while (channel.IsOpen)
{
var success = subscription.Next(5000, out BasicDeliverEventArgs eventArgs);
if (success == false) continue;
messageBody = eventArgs.Body;
channel.BasicAck(eventArgs.DeliveryTag, false);
}
}
return messageBody;
}
In the above code, there are two issues (can be more).
1) Even after writing prefetchCount = 1, it still reads all the messages.
2) Ever after waiting for 5 seconds, I never get a success and I'm not able to send the body to outside.
I have written one more code, which does the same thing but in the post itself, it was written that it is not recommended way of doing.
Sample Code:
using (var signal = new ManualResetEvent(false))
{
var consumer = new EventingBasicConsumer(channel);
consumer.Received +=
(sender, args) =>
{
messageBody = args.Body;
signal.Set();
};
//// start consuming
channel.BasicConsume(containerName, true, consumer);
// wait until message is received or timeout reached
bool timeout = !signal.WaitOne(TimeSpan.FromSeconds(10));
// cancel subscription
channel.BasicCancel(consumer.ConsumerTag);
if (timeout)
{
// timeout reached - do what you need in this case
throw new Exception("timeout");
}
return messageBody;
// at this point messageBody is received
}
Try reading with channel.BasicGet instead, something like this:
private byte[] ReadRabbitMsg(IModel channel, string queue)
{
if (channel.MessageCount(queue) == 0) return null;
BasicGetResult result = channel.BasicGet(queue, true);
if (result == null) return null;
else
{
IBasicProperties props = result.BasicProperties;
byte[] buff = result.Body;
}
}

RabbitMQ and SharedQueue closed

Im using RabbitMQ to send simple short int information, first I'm sending id to one project like that:
private void SendPgcIdToRabbitMQ(string id)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
bool durable = true;
channel.QueueDeclare("XQueue", durable, false, false, null);
var body = Encoding.UTF8.GetBytes(id);
channel.BasicPublish("", "XQueue", null, body);
Console.WriteLine(" [x] Sent {0}", id);
}
}
}
and listener of it:
public void Listener()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("XQueue", true, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("XQueue", false, consumer);
Console.WriteLine(" [*] Waiting for messages. " +
"To exit press CTRL+C");
while (true) {
var ea =
(BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
AddPGCFileID(message);
channel.BasicAck(ea.DeliveryTag, false);
Thread.Sleep(500);
}
}
}
}
It works fine, so after receiving message I'm processing some operation wit it, then I get second ID and create other queue to do this same:
private void SendSurveyIdToRabbitMQ(int yID)
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection()) {
using (var channel = connection.CreateModel()) {
bool durable = true;
channel.QueueDeclare("YQueue", durable, false, false, null);
var body = Encoding.UTF8.GetBytes(yID.ToString());
channel.BasicPublish("", "YQueue", null, body);
Console.WriteLine(" [x] Sent {0}", yID);
}
}
}
and receive:
public void InquiryListener()
{
var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection()) {
using (var channel = connection.CreateModel()) {
channel.QueueDeclare("YQueue", true, false, false, null);
var consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("YQueue", false, consumer);
Console.WriteLine(" [*] Waiting for messages. " +
"To exit press CTRL+C");
while (true) {
var ea =
(BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine(" [x] Received {0}", message);
StartProcessing(Convert.ToInt32(message));
channel.BasicAck(ea.DeliveryTag, false);
Thread.Sleep(500);
}
}
}
}
First queue sending and receiving works fine but at second I get:
It is strange because it was working that way, from some time I'm geting this problem. I whas reseting rabbitmq, removin all queues etc. can't find where is a problem. Any ideas?
edit:
I whas debuging to know if second process is ending properly (eariel crash on second proces don't cause problem with rabbitmq) and it passed, I whas supriced because no error ocurs on YQueue, but after about minute of working my process (only waiting, non incomming message, non processing) I gey this same exception on XQueue
Check first if the queue is empty before executing while(true){ ... }.

Categories

Resources