Develop Reference

C# RabbitMQ why is the second worker not picking up work?

I am trying to understand all the RabbitMQ options what I think I want its just a worker queue so I have one queue and workers just take off one item and process it.
I create a new Direct Exchange (I think that is right!?)
Firstly I would like to know why in this example, I add 4 new messages to the exchange/queue. I don't start any workers. I then start the first worked and then the second, but the second one does not process any and the first worked processes them all!?
What am I doing wrong, why does this not work? please find the full example code below.
I also don't seem to have the publish confirms working right, as only sometimes on the right most output does it say "Message Acknowledged..."
I have read https://www.rabbitmq.com/tutorials/tutorial-two-dotnet.html and gone through the other pages that follow, but it's not hugely clear.
The emitter:
using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Text;
using RabbitMQ.Client;
namespace PublishConfirms.Emit
{
class Program
{
public static ConcurrentDictionary<ulong, string> _outstandingConfirms = new ConcurrentDictionary<ulong, string>();
static void Main(string[] args)
{
var factory = new ConnectionFactory
{
HostName = "localhost"
};
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
// We need to enable published confirms on the channel
channel.ConfirmSelect();
channel.ExchangeDeclare(exchange: "DirectExchange",
type: ExchangeType.Direct);
var queueName = "DirectExchangeQueue";
// Make sure to create the queue in case it doesn't exits
channel.QueueDeclare(queue: queueName,
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
channel.BasicAcks += (sender, ea) =>
{
Console.WriteLine("Message Acknowledged with Delivery Tag {0}", ea.DeliveryTag);
// message is confirmed
CleanOutstandingConfirms(ea.DeliveryTag, ea.Multiple);
};
channel.BasicNacks += (sender, ea) =>
{
// message is nack-ed (messages that have been lost
_outstandingConfirms.TryGetValue(ea.DeliveryTag, out string body);
Console.WriteLine($"Message with body {body} has been nack-ed. Sequence number: {ea.DeliveryTag}, multiple: {ea.Multiple}");
CleanOutstandingConfirms(ea.DeliveryTag, ea.Multiple);
};
var message = "A message here";
var body = Encoding.UTF8.GetBytes(message);
_outstandingConfirms.TryAdd(channel.NextPublishSeqNo, message);
var properties = channel.CreateBasicProperties();
properties.Persistent = true;
channel.BasicPublish(exchange: "",
routingKey: queueName,
basicProperties: properties,
body: body);
Console.WriteLine("Sent message '{0}'", message);
}
}
Console.WriteLine("Press any key to exit");
Console.ReadLine();
}
static void CleanOutstandingConfirms(ulong sequenceNumber, bool multiple)
{
if (multiple)
{
var confirmed = _outstandingConfirms.Where(k => k.Key <= sequenceNumber);
foreach (var entry in confirmed)
{
_outstandingConfirms.TryRemove(entry.Key, out _);
}
}
else
{
_outstandingConfirms.TryRemove(sequenceNumber, out _);
}
}
}
}
The worker/receiver
using System;
using System.Text;
using System.Threading;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
namespace PublishConfirms.Receive
{
class Program
{
// We can run multiples of these and only one will get a messages from the queue with no sharing
static void Main(string[] args)
{
var factory = new ConnectionFactory
{
HostName = "localhost"
};
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
// This will create the exchange if needed
channel.ExchangeDeclare(exchange: "DirectExchange",
type: ExchangeType.Direct);
var queueName = "DirectExchangeQueue";
// Make sure to create the queue in case it doesn't exits
channel.QueueDeclare(queue: queueName,
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
channel.QueueBind(queue: queueName,
exchange: "DirectExchange",
routingKey: ""); // We keep the routing key the same as we dont want different handlers
// If we were to have different routes then we
// would most probably have to create a random queue e.g.
// var queueName = channel.QueueDeclare().QueueName;
Console.WriteLine("Waiting for messages...");
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
Console.WriteLine("Received message: {0}", message);
Console.WriteLine("Processing...");
Thread.Sleep(3000); // simulate some work
Console.WriteLine("Processing Complete");
// send an acknowledgement back
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
};
channel.BasicConsume(queue: queueName,
autoAck: false,
consumer: consumer);
Console.WriteLine("Press any key to exit");
Console.ReadLine();
}
}
}
}
}

Your problem lays in understanding how the Direct exchange works, and probably the queue binding keys and routing keys as well.
A Direct exchange deliveries its messages to the queues whose binding key exactly matches the routing key of the message.
The binding key is the key the queue is binding to the exchange by.
The routing key is the key the message sent is routed by
Therefore the problem with your code is either:
With publishing your messages:
...
channel.BasicPublish(exchange: "",
routingKey: "", //
basicProperties: properties,
body: body);
...
Or you could change the binding key when creating the queue:
...
channel.QueueBind(queue: queueName,
exchange: "DirectExchange",
routingKey: queueName); // As you stated in your code that
// the queue name is your bindingKey
...
You can choose whichever one you prefer.
Other than that, when you publish your message you do it to an empty exchange name,
it should be:
channel.BasicPublish(exchange: "DirectExchange", // As you stated at the start of your code
routingKey: queueName,
basicProperties: properties,
body: body);
I'd suggest reading up more about RabbitMQ and understanding more of the concepts about their implementation.
You may want to see references about:
Fanout Exchange
Routing
I hope that helped.

After reading what itama said I had another look and firstly the BasicPublish exchange was an empty string so that would have not helped! I decided against using the routing key as the queue name and kept it as an empty string.
For me I wanted an exchange with a single queue and then multiple workers that would handle one item from the queue at a time.
The issue with it only being handled by one worked seemed to be resolved when I added in this piece of code on the worker.
channel.BasicQos(0, 1, false);
After adding this the other workers got a message each. I ended up adding 50-100 messages to the queue and then started 10 workers, they all got a queue item and processed it which was nice to see.
For the Publish Confirm BasicAcks and BasicNacks callbacks I found this link https://rianjs.net/2013/12/publisher-confirms-with-rabbitmq-and-c-sharp this states that we need to call the channel.WaitFormConfirmsOrDie() after making the BasicPublish call, previously my code was not waiting for an acknowledgement.
The full code is as follows:
The emitter/producer:
using System;
using System.Collections.Concurrent;
using System.Linq;
using System.Text;
using RabbitMQ.Client;
namespace PublishConfirms.Emit
{
class Program
{
public static ConcurrentDictionary<ulong, string> _outstandingConfirms = new ConcurrentDictionary<ulong, string>();
static void Main(string[] args)
{
var factory = new ConnectionFactory
{
HostName = "localhost"
};
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
// -------------------------------------------------------------------
// Setup the Exchange and Queue then bind the queue to the exchange
// -------------------------------------------------------------------
// This will create the exchange if needed
channel.ExchangeDeclare(exchange: "DirectExchange",
type: ExchangeType.Direct);
var queueName = "DirectExchangeQueue";
// Make sure to create the queue in case it doesn't exits
channel.QueueDeclare(queue: queueName,
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
channel.QueueBind(queue: queueName,
exchange: "DirectExchange",
routingKey: "");
// -------------------------------------------------------------------
// Setup Publish Confirms
// -------------------------------------------------------------------
channel.BasicAcks += (sender, ea) =>
{
Console.WriteLine("Message Acknowledged with Delivery Tag {0}", ea.DeliveryTag);
// message is confirmed
CleanOutstandingConfirms(ea.DeliveryTag, ea.Multiple);
};
channel.BasicNacks += (sender, ea) =>
{
// message is nack-ed (messages that have been lost)
_outstandingConfirms.TryGetValue(ea.DeliveryTag, out string body);
Console.WriteLine($"Message with body {body} has been nack-ed. Sequence number: {ea.DeliveryTag}, multiple: {ea.Multiple}");
CleanOutstandingConfirms(ea.DeliveryTag, ea.Multiple);
};
// We need to enable published confirms on the channel
channel.ConfirmSelect();
// -------------------------------------------------------------------
// Setup The message and add it to the ConcurrentDictionary, so we can
// Remove it when the BasicAcks is called
// -------------------------------------------------------------------
var message = "YOUR MESSAGE HERE";
var body = Encoding.UTF8.GetBytes(message);
var nextPublishSequenceNo = channel.NextPublishSeqNo;
Console.WriteLine("Next Publish Sequenece Number: {0}", nextPublishSequenceNo);
_outstandingConfirms.TryAdd(nextPublishSequenceNo, message);
// Make sure the message is written to disk as soon as it reaches the queue
// Imagine this will be slower but safer, this is also stored in memory if there is no memory pressure!
var properties = channel.CreateBasicProperties();
properties.Persistent = true;
channel.BasicPublish(exchange: "DirectExchange",
routingKey: "",
basicProperties: properties,
body: body);
channel.WaitForConfirmsOrDie();
Console.WriteLine("Sent message '{0}'", message);
}
}
}
static void CleanOutstandingConfirms(ulong sequenceNumber, bool multiple)
{
if (multiple)
{
var confirmed = _outstandingConfirms.Where(k => k.Key <= sequenceNumber);
foreach (var entry in confirmed)
{
_outstandingConfirms.TryRemove(entry.Key, out _);
}
}
else
{
_outstandingConfirms.TryRemove(sequenceNumber, out _);
}
}
}
}
The receiver/worker:
using System;
using System.Text;
using System.Threading;
using RabbitMQ.Client;
using RabbitMQ.Client.Events;
namespace PublishConfirms.Receive
{
class Program
{
// We can run multiples of these and only one will get a messages from the queue with no sharing
static void Main(string[] args)
{
var factory = new ConnectionFactory
{
HostName = "localhost"
};
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
// -------------------------------------------------------------------
// Setup the Exchange and Queue then bind the queue to the exchange
// -------------------------------------------------------------------
// This will create the exchange if needed
channel.ExchangeDeclare(exchange: "DirectExchange",
type: ExchangeType.Direct);
var queueName = "DirectExchangeQueue";
// Make sure to create the queue in case it doesn't exits
channel.QueueDeclare(queue: queueName,
durable: true,
exclusive: false,
autoDelete: false,
arguments: null);
// Only dispatch one message at a time to a worked and wait for its acknowledgement
// Adding this in seems to correct the issue where only one worker would pick
// up the queued items
channel.BasicQos(0, 1, false);
// We keep the routing key the same as we don't want different handlers
// If we were to have different routes then we
// would most probably have to create a random queue e.g.
// var queueName = channel.QueueDeclare().QueueName;
channel.QueueBind(queue: queueName,
exchange: "DirectExchange",
routingKey: "");
Console.WriteLine("Waiting for messages...");
var consumer = new EventingBasicConsumer(channel);
consumer.Received += (model, ea) =>
{
var body = ea.Body.ToArray();
var message = Encoding.UTF8.GetString(body);
Console.WriteLine("Received message: {0}", message);
Console.WriteLine("Processing...");
Thread.Sleep(3000); // simulate some work
Console.WriteLine("Processing Complete");
// send an acknowledgement back
channel.BasicAck(deliveryTag: ea.DeliveryTag, multiple: false);
};
channel.BasicConsume(queue: queueName,
autoAck: false,
consumer: consumer);
Console.WriteLine("Press any key to exit");
Console.ReadLine();
}
}
}
}
}

ASP.NET MVC - View is not returned until asynchonous emails sending is completed

I have a Contact page where users are able to complete a form which is sent to a Controller.
This Controller generates an email based on form data and returns a success view to the user.
The problem is that the success view is not displayed until email sending is completed, but this takes too much time (around 30 seconds), which is not acceptable.
First, I tried synchronous sending:
public ActionResult Contact(string TestMessage)
{
// Prepare SMTP client
SmtpClient Client = new SmtpClient()
{
DeliveryMethod = SmtpDeliveryMethod.Network,
EnableSsl = true,
Host = "smtp.test.com",
Port = 587,
UseDefaultCredentials = false,
Credentials = new System.Net.NetworkCredential("smtpuser", "smtppass")
};
// Prepare message
MailMessage MailMessage = new MailMessage()
{
From = new MailAddress("sender#mydomain.com"),
Subject = "Test",
BodyEncoding = System.Text.Encoding.UTF8,
SubjectEncoding = System.Text.Encoding.UTF8
};
MailMessage.To.Add(new MailAddress("recipient#mydomain.com"));
MailMessage.Body = TestMessage;
// Send mail
Client.Send(MailMessage);
// Return success view
return View("ContactSuccess");
}
After that I tried asynchonous sending. I have spent hours reading StackOverflow questions to find the miracle solution. But the result is always the same, page is still waiting for email sending to be completed.
Asynchronous attempt 1
// Send mail
Client.SendCompleted += (s, e) =>
{
Client.Dispose();
MailMessage.Dispose();
};
System.Threading.ThreadPool.QueueUserWorkItem(o =>
Client.SendAsync(MailMessage, Tuple.Create(Client, MailMessage)));
Asynchronous attempt 2
System.Threading.Tasks.Task.Run(() => SendViaAsync(MailMessage));
Asynchronous attempt 3
System.Web.Hosting.HostingEnvironment.QueueBackgroundWorkItem(
cancellationToken => Client.Send(MailMessage)
);
Asynchronous attempt 4
// Send mail
System.Threading.ThreadPool.QueueUserWorkItem(SendViaAsync, MailMessage);
With added methods:
private void SendViaAsync(object MailMessageObject)
{
MailMessage MailMessage = (MailMessage)MailMessageObject;
SmtpClient Client = Utilities.CreateEmailClient();
Client.SendCompleted += new SendCompletedEventHandler(SmtpClient_SendCompleted);
Client.Send(MailMessage);
}
void SmtpClient_SendCompleted(object sender, System.ComponentModel.AsyncCompletedEventArgs e)
{
MailMessage mail = e.UserState as MailMessage;
if (!e.Cancelled && e.Error != null)
{
// error handling
}
}
Asynchronous attempt 5
public async System.Threading.Tasks.Task<ActionResult> Contact(string TestMessage)
// ...
// Send mail
await SendEmail(MailMessage);
With added method:
public async System.Threading.Tasks.Task SendEmail(MailMessage MailMessage)
{
SmtpClient Client = new SmtpClient()
{
DeliveryMethod = SmtpDeliveryMethod.Network,
EnableSsl = true,
Host = "smtp.test.com",
Port = 587,
UseDefaultCredentials = false,
Credentials = new System.Net.NetworkCredential("smtpuser", "smtppass")
};
await Client.SendMailAsync(MailMessage);
}
Please note that when locally testing, email is sent immediately (in both synchronous or asynchronous modes). In production only, it takes around 30 seconds. So maybe there is something like an antivirus scan in my hosting provider, but anyway I can't figure out why the success view is not sent until mail sending is completed.
I am desperate. Any help would be much appreciated.

Any attempts with await will definitely fail to accomplish what you want by definition. But in general I don't think it's wise to have an asp.net page starting up processes which may keep running beyond the lifetime of the request. And perhaps that's exactly what's happening internally: ASP.Net not finalising the request until all its child processes have finished.
Personally, I would de-couple the process altogether. First because each part will be simpler to maintain, but also because it's more secure.
Have your application dump the email, or the data to build the email, into a queue of sorts. This could be a database, a file, a folder or whatever, but some temporary storage.
Then develop a background service or daemon, which reads this queue and processes the emails. It will end up being more flexible, more scalable and safer.

I think I have understood the problem. I can't believe it.
I didn't mention it in my bug reproduction, but I am using Google reCAPTCHA in the contact form.
It seems that reCAPTCHA is very slow in production environment!
After deactivating it, it appears that email is immediately sent.
Now, I just have to replace reCAPTCHA with a faster checking mecanism.
Sorry for disturbance, I hope this will help some of us later. At least some asynchronous programming examples are listed here.

There is a builtin functionality in asp net you can use it as easy as written below
public ActionResult SendEmail(User user)
{
HostingEnvironment.QueueBackgroundWorkItem(ct =>
{
// Prepare SMTP client
SmtpClient Client = new SmtpClient()
{
DeliveryMethod = SmtpDeliveryMethod.Network,
EnableSsl = true,
Host = "smtp.test.com",
Port = 587,
UseDefaultCredentials = false,
Credentials = new System.Net.NetworkCredential("smtpuser", "smtppass")
};
// Prepare message
MailMessage MailMessage = new MailMessage()
{
From = new MailAddress("sender#mydomain.com"),
Subject = "Test",
BodyEncoding = System.Text.Encoding.UTF8,
SubjectEncoding = System.Text.Encoding.UTF8
};
MailMessage.To.Add(new MailAddress("recipient#mydomain.com"));
MailMessage.Body = TestMessage;
// Send mail
Client.Send(MailMessage);
});
// Return success view
return View("ContactSuccess");
}
if you want to go over advance scenarios , like handling many background jobs you can use Hangfire library
this blog post will also help you solve your problem

Strategy for sending async emails in a sync method in ASP.Net MVC

I'm looking for help on the strategy for sending emails after a student has registered for an event/class. The registering occurs in a synchronous method called from my controller but after I've registered (added the row to the db) for that event I am sending out an email. Sendgrid sends this email out from a async method and I'm not sure what the best way to handle this is? I have spinners and other items on the screen to inform the user that the process of registering is complete, do I make the entire call to regsiter async or is there a better way? Ideally I would like to keep the method 'SaveYogaSpaceEvent' sync and just make the sending of the email async.
Here is my code
Here is the method in my controller that registers the student.
Tuple<int, string, string, string> eventResults = _yogaSpaceService.SaveYogaSpaceEvent(User.Identity.GetUserId(), details);
Here is the method
internal Tuple<int, string, string, string> SaveYogaSpaceEvent(string userId, EventDetails details)
{
// register in the db
// do some other stuff
// create email templates
// send emails
_emailService.SendEmail(new IdentityMessage { Destination = yogaProfile.ApplicationUser.Email, Subject = eventEmail.Item1, Body = eventEmail.Item2 }, eventEmail.Item3);
}
Here is what the 'sendemail' method look like
public void SendEmail(IdentityMessage message, List<Attachment> attachments)
{
ConfigSendGridAsync(message, attachments);
}
private async void ConfigSendGridAsync(IdentityMessage message, List<Attachment> attachments)
{
// configure the client
var client = new SendGridClient(apiKey);
var msg = MailHelper.CreateSingleEmailToMultipleRecipients(from, emailAddresses, subject, plainTextContent, htmlContent);
var response = await client.SendEmailAsync(msg);
}

RabbitMQ: Sending messages from one pc to another

I am trying to get a rabbitMQ queue set up that sits on one pc, and recieves messagess from other computers giving it tasks. I have followed all the tutorials on the rabbit website but these only apply to local host. Can someone explain how I get this same code to communicate across 2 computers, not just from the same computer.
I have the following code:
Sender.cs
class Send
{
static void Main(string[] args)
{
Console.WriteLine("------------------");
Console.WriteLine("RabbitMQ Test");
Console.WriteLine("------------------");
var factory = new ConnectionFactory() { HostName = "localHost" };
try
{
using (var connection = factory.CreateConnection())
{
using (var channel = connection.CreateModel())
{
channel.QueueDeclare("abc", false, false, false, null);
Console.WriteLine("Enter the messages you want to send (Type 'exit' to close program)...");
string message = null;
while (message != "exit")
{
message = Console.ReadLine();
var body = Encoding.UTF8.GetBytes(message);
channel.BasicPublish("", "abc", null, body);
}
}
}
}
catch (Exception e)
{
string message = e.ToString();
}
}
Reciever.cs
class Recieve
{
static void Main(string[] args)
{
ConnectionFactory factory = new ConnectionFactory()
{
HostName = "localhost"
};
using (IConnection connection = factory.CreateConnection())
{
using (IModel channel = connection.CreateModel())
{
channel.QueueDeclare("abc", false, false, false, null);
QueueingBasicConsumer consumer = new QueueingBasicConsumer(channel);
channel.BasicConsume("abc", true, consumer);
Console.WriteLine(" [*] Waiting for messages." +
"To exit press CTRL+C");
while (true)
{
var ea = (BasicDeliverEventArgs)consumer.Queue.Dequeue();
var body = ea.Body;
var message = Encoding.UTF8.GetString(body);
Console.WriteLine("[Recieved]: {0}", message);
}
}
}
}
}
Is the idea to get these communicating across 2 computers to change the ConnectionFactory's hostname to the IP of the other computer or something to that extent? I have installed rabbit correctly on both computers, and this code runs correctly on each computer individually. I just need the communication to happen across the computers.
Any help would be greatly appreciated. I can't find any examples of this anywhere on the internet.

RabbitMQ is a centralized message queue. You would only install it on your server machine (or a cluster of server machines), and not on each publishing or subscribing client. Clients would then subscribe or publish to queues on the centralized machine.
In your development environment you should decide which of your two machines you want to act as the server. Then pass that hostname and password each client. Both clients should be working towards the same hostname.
var server = "http://127.0.0.1"; // An install of RabbitMQ reachable from both clients
var password = "guest";
var username = "guest";
var connectionFactory = new ConnectionFactory { HostName = server, Password = password , Username = username};
If you want to do message passing without installing something on a server you should take a look at ZeroMQ

You can check out shovel plugin - it is built to take messages messages from queue on one node of rabbit and publishes these messages on other one, while taking care of poor networking (WAN).
Check the official description

Background Task to send an email

MVC 5.1, .NET 4.5
In our simple web site we need to send an email. Our email service is called by a Post back event and it sends the email, then builds a result page and returns it.
This slows the page since the page has to wait for the synchronous email to be sent.
If we run the email in a different thread, can the site continue on to create the page and return it to the user as the email is being processed, or will a long running task cause the page to be not rendered.
Basically, can I send am email non blocking for performance reasons.
Note: We can't send the email request to a service or other system.

Assuming that you you use built in SmtpClient you can use SmtpClient.SendMailAsync method
smtpClient.SendMailAsync(new MailMessage{ ... });
Sends the specified message to an SMTP server for delivery as an asynchronous operation

Calling StartNew is functionally equivalent to creating a task by using one of its constructors, and then calling the Task.Start method to schedule the task for execution.
Task.Factory.StartNew(() => {
//do somethings
});
Example:
Task.Factory.StartNew(() => {
SendMail("Hello world", "email#yahoo.com", "TEST");
});
public static bool SendMail(string subject, string to, string body)
{
string fromMailAddress = ConfigurationManager.AppSettings["MailAddress"];
string fromMailPassword = ConfigurationManager.AppSettings["MailPassword"];
string fromMailName = ConfigurationManager.AppSettings["MailName"];
var networkConfig = new NetworkCredential(fromMailAddress, fromMailPassword);
var mailServer = new SmtpClient()
{
Host = ConfigurationManager.AppSettings["SmtpHost"],
UseDefaultCredentials = false,
Credentials = networkConfig
};
if (!string.IsNullOrEmpty(ConfigurationManager.AppSettings["SmtpPort"]))
mailServer.Port = Convert.ToInt32(ConfigurationManager.AppSettings["SmtpPort"]);
var message = new MailMessage()
{
Subject = subject,
SubjectEncoding = Encoding.UTF8,
IsBodyHtml = true,
BodyEncoding = Encoding.UTF8,
};
//message send config
message.To.Add(new MailAddress(to));
message.From = new MailAddress(fromMailAddress, fromMailName);
message.Body = body;
try
{
mailServer.SendAsync(message, null);
}
catch (Exception ex)
{
return false;
}
return true;
}