Develop Reference

Correct pattern for ZeroMQ Server/Client type

I have a server that needs to get instructions to run processes for clients on another machine.
The clients send a job message, the Server processes the job and later sends the back results.
I tried using the NetMQ Request-Response pattern (see below)
This works nicely for 1 client, BUT if a second client sends a request before previous client job is finished - I get an error.
I really need to be able to receive ad-hoc messages from clients, and send results when they are completed. Clearly, I am using the wrong pattern, but reading the ZeroMQ docs has not highlighted a more appropriate one.
namespace Utils.ServerMQ
{
class ServerMQ
{
public static void Go()
{
using (var responseSocket = new ResponseSocket("#tcp://*:393"))
{
while (true)
{
Console.WriteLine("Server waiting");
var message = responseSocket.ReceiveFrameString();
Console.WriteLine("Server Received '{0}'", message);
//System.Threading.Thread.Sleep(1000);
var t2 = Task.Factory.StartNew(() =>
{
RunProcMatrix(message, responseSocket);
});
}
}
}
public static void RunProcMatrix(object state, ResponseSocket responseSocket)
{
var process = new Process
{
StartInfo = new ProcessStartInfo
{
FileName = Path.Combine(#"H:\Projects\Matrix\Matrix\bin\Debug\", "Matrix001.exe"),
Arguments = (string)state,
WindowStyle = ProcessWindowStyle.Normal,
CreateNoWindow = false
}
};
process.Start();
process.WaitForExit();
responseSocket.SendFrame((string)state);
}
}
}

You want a ROUTER socket on the server side, so it can receive multiple requests at a time. (Guide) REQ sockets on the client side are still fine unless the server may arbitrarily push data to them, then they need to be DEALER sockets.
Note that for sockets beyond REQ/RESP you need to manually handle the message envelope (the first frame of the message indicating its destination). Guide
The 0MQ docs are incredibly dense... I don't blame you for not intuiting this from them :)
This example from the NetMQ docs is full ROUTER-DEALER: https://netmq.readthedocs.io/en/latest/router-dealer/#router-dealer, you can take just the router side and it should work the same though.

How to speed up sending of e-mails?

I need to send different e-mails to different people. A different body of the e-mail is generated for every recipient and then I call the following method. That is, if there are 10 recipients, the following method is called 10 times:
public int Send_Mail(string send_from_bc, string send_to_bc, string subject_bc, string body_bc)
{
try
{
int count_returned = 0;
var send_from = new MailAddress(send_from_bc, "The Admin");
var send_to = new MailAddress(send_to_bc);
string subject = subject_bc;
string body = body_bc;
var smtp = new SmtpClient
{
Host = "**my smtp host**",
Port = 25,
DeliveryMethod = SmtpDeliveryMethod.Network,
Timeout = 300000
};
using (var message = new MailMessage(send_from, send_to)
{
Subject = subject,
Body = body
})
{
message.IsBodyHtml = true;
smtp.Send(message);
count_returned += 1;
}
return count_returned;
}
catch (Exception ex)
{
throw new Exception("Exception caught in Send_Mail", ex);
}
}
The problem is that this is taking an incredible amount of time to send even a few mails (like 10 or 15 mails). It takes around 2-5 minutes to send a single mail. Now, I do believe that partly the SMTP host provided to me is at fault. But what can I do to speed things up?
Also, is there some way to achieve this: Right now, the user clicks on a button and then say, for sending two mails, he/she has to wait 2-5 minutes (sometimes even more) before control is returned and the value of a label is changed to "E-mails sent.". Can something be done so that the user just clicks on the button, and the e-mail sending is initiated and he/she could just close the window and get on with his work?
I'm stuck on a critical stage right now and any help would be appreciated.

First thing first - don't make the user wait for this action. The session could well time out if you have to wait 2-5 minutes. Instead make this action an asynchronous task that runs in the background.
I'd then consider popping up a notification stating that the emails have been sent to the user later on, rather than forcing them to wait.
A starter for 10 - move your sending email code into a separate class:
public class EmailSender
{
public EmailSender(/* Parameters required, e.g. list of emails */)
{ }
public void SendEmails()
{
// Long running task
}
}
Then in your page, say a button is clicked:
protected void btn_SendEmails_Clicked(object sender, EventArgs e)
{
EmailSender emailSender = new EmailSender(/* Any setup code required */);
ThreadPool.QueueUserWorkItem(emails => emailSender.SendEmails());
}
To handle the notifications back to the user you'll want to raise an Event when the emails have finished and probably using something like SignalR to alert the browser.
Note that I've not tried compiling this code but should give you the general idea.

Multithreaded loop with rabbitMQ consumer and WebApi call - WebApi response never returned

I've got a multithreaded Windows service which is consuming messages off a Rabbit queue and sending emails based on the content of the messages.
When the rabbit client is initialized at startup, it limits the Threadpool threads with a Min and Max value.
For each message taken off the queue the service is sending an HTTP request to a web api service using HttpClient GetAsync method to retrieve email address.
The problem is that the request goes off to the data service, but the response never comes back. The windows service keeps consuming messages off the queue and hangs after a while (probably runs of of free threads) - it's waiting for any of the calls to web api to complete which they never do.
I was able to resolve the problem using a Semaphore class for the Rabbit loop rather than trying to limit the Threadpool directly, however, I'd like to know why the service got into this state in the first place. Is that to do with the GetAsync call? Is it perhaps freeing up the thread for the duration of the request, so that the main loop can steal it for a next request?
Any ideas?
The original loop:
while (!_stopped)
{
if (_paused) continue;
try
{
using (var messageBusReceiver = _rabbitQueueClient.ConfigureMessageBusReceiver())
{
using (_consumer = messageBusReceiver.Listen<PublishableItem>())
{
while (!_stopped)
{
if (_paused) continue;
_consumer.Consume(callback, consumeSynchronously: false);
_communicationErrorCount = 0;
}
}
}
}
The Consume method is eventually doing this:
_threadPoolProvider.QueueUserWorkItem(o =>
consumeMessage(callback, eventArgs, o), message);
The callback begins with the following lines - the null checking line is never reached:
var foo = _fooService.GetFoo(messageInfo.FooId);
if (foo == null)
{
throw new FooNotFoundException(
String.Format(CultureInfo.InvariantCulture, "Foo was not found for FooId of {0}", messageInfo.FooId));
}
The client method:
public Foo GetFoo(Guid id)
{
var path = getPathWithQueryStringAndDebug("getfoo", "id", id.ToString());
var response = _client.GetAsync(path).Result;
return processResponse<FooDto>(response);
}

Sending mail periodically in ASP.net

I'm trying to send confirmation mails to users periodically in ASP.NET.
To do this I polulate a queue with mails and check it every 30 seconds. Any confirmation emails in the queue at this time are sent and then cleared from the queue.
Does anyone know how to do this?
Here is my sending mail code
public static bool SendMail(string AdminMail,string AdminPassword,string subject,string toAddress, string content,DateTime SendTime)
{
toAddressListProperty.Enqueue(toAddress);
if(date==null)
{
date = DateTime.Now.Second;
}
if (date-SendTime.Second > 120)
{
var message = new MailMessage
{
From = new MailAddress(AdminMail)
};
foreach (var toAddressl in toAddressListProperty)
{
message.To.Add(new MailAddress(toAddressl));
}
message.Subject = subject;
message.Body = content;
message.IsBodyHtml = true;
var smtp = new SmtpClient
{
Credentials = new System.Net.NetworkCredential(AdminMail, AdminPassword),
Port = 587,
Host = "smtp.gmail.com",
EnableSsl = true
};
smtp.Send(message);
//date = SendTime;
return true;
}
return false;
}

I have done this using a background thread. I did a little research, and I believe this is an ok approach. There are a few dangers, which this blog details.
The main thing is to ensure you never throw an exception from a background thread, as I believe that will cause the web process to restart. Also, incase the thread dies, I ensure it is running on every call.
I have been using this approach for a few months, and so far no issues.
Also I run it every 1 second, this minamizes the amount of time you might loose emails due to an app shutdown.
public class BackgroundSmtpService
{
private ILog _log = LogManager.GetLogger(typeof(BackgroundSmtpService));
private readonly SmtpService SmtpService;
private static Thread _watchThread;
private static List<Email> _emailToSend = new List<Email>();
public BackgroundSmtpService(SmtpService smtpService)
{
SmtpService = smtpService;
}
public void Send(Email email)
{
lock (_emailToSend)
{
_emailToSend.Add(email);
}
EnsureRunning();
}
private void EnsureRunning()
{
if (_watchThread == null || !_watchThread.IsAlive)
{
lock (SmtpService)
{
if (_watchThread == null || !_watchThread.IsAlive)
{
_watchThread = new Thread(ThreadStart);
_watchThread.Start();
}
}
}
}
private void ThreadStart()
{
try
{
while (true)
{
Thread.Sleep(1000);
try
{
lock (_emailToSend)
{
var emails = _emailToSend;
_emailToSend = new List<Email>();
emails.AsParallel().ForAll(a=>SmtpService.Send(a));
}
}
catch (Exception e)
{
_log.Error("Error during running send emails", e);
}
}
}
catch (Exception e)
{
_log.Error("Error during running send emails, outer", e);
}
}
}

You might want to consider using Quartz.net library. It have decent documentation and it's fairly easy to use.

The biggest challenge you'll have with this is that any time your application pool recycles it will take a new request to kick stats your "timer". If you had an HTTP monitor application such as Pingdom to poll your server it shouldn't be a problem, but then again you could also just use a third party monitor tool to hit your a page on your site every N seconds that would send out the mail and issue a response.
I myself would use a Windows service to pull a queue from a database and send out messages that way.

Easiest way is to create a VBScript that sends an HTTP GET request to http://localhost/SendConfirmationEmails.aspx
You'd start the VBScript in your global.asax Application_Start method.
The SendConfirmationEmails.aspx would act as a simple web service (you could use an ashx, or actual web service asmx if you wanted). It would only be accessible on the localhost so remote users wouldn't be able to spam it.
Using a windows service is probably the best practice method, but a simple VBScript will get the job done.
surl="http://localhost/SendConfirmationEmails.aspx"
set oxmlhttp=createobject("msxml2.xmlhttp")
with oxmlhttp
.open "GET",surl,false
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send srequest
end with
You'd put the code above in a while wend loop with a Sleep to delay every 30 seconds...

Invalid Operation Exception In Calling Async Method from inside a Parallel.ForEach Loop

I have inherited a windows service that processes a large number of e-mails in a queue. Sounds simple, Grab queue, send e-mail, if SmtpClient.SendAsync does not return an error from the call back then flag the e-mail in the DB as being sent.. I am using a Semaphore to waitone on the thread so multiple calls can be made to the Async Send method of the SMTP Client. This is the only way I can get the status and per Microsoft docs it has to finish the operation before another call can be made async. So now for the fun part. I decided to use a Parallel.ForEach to get he queue like so. This method is called in the Windows Service OnStart. Please note I have tried calling this method on a separate Thread and get the same results.
I am thinking that either A, I am missing something obvious, due to my lack of knowledge on threading, or something is flat bugged. Most likely A.
private static void ProcessEmailQueue()
{
List<EmailQueue> emailQueue =
_repository.Select<EmailQueue>().Where(x => x.EmailStatuses.EmailStatus == "Pending").ToList();
Parallel.ForEach(emailQueue, message =>
{
_smtpMail.FromAddress = message.FromAddress;
_smtpMail.ToAddress = message.ToAddress;
_smtpMail.Subject = message.Subject;
_smtpMail.SendAsHtml = message.IsHtml > 0;
_smtpMail.MessageBody = message.MessageBody;
_smtpMail.UserToken = message.EmailQueueID;
bool sendStatus = _smtpMail.SendMessage();
// THIS BLOWS UP with InvalidOperation Exception
});
}
Here is the SMTP Method being called from withing the loop.
public bool SendMessage()
{
mailSendSemaphore = new Semaphore(0, 10); // This is defined as private static Semaphore mailSendSemaphore;
try
{
var fromAddress = new MailAddress(FromAddress);
var toAddress = new MailAddress(ToAddress);
using (var mailMessage = new MailMessage(fromAddress, toAddress))
{
mailMessage.Subject = Subject;
mailMessage.IsBodyHtml = SendAsHtml;
mailMessage.Body = MessageBody;
Envelope = mailMessage;
smtp.SendCompleted += smtp_SendCompleted;
smtp.SendAsync(mailMessage, UserToken);
mailSendSemaphore.WaitOne();
return _mailSent;
}
}
catch (Exception exception)
{
_logger.Error(exception);
return _mailSent;
}
}
CALLBACK For Smtp Send
private void smtp_SendCompleted(object sender, AsyncCompletedEventArgs e)
{
if (e.Cancelled)
{
}
if (e.Error != null)
{
}
else
{
_mailSent = true;
}
mailSendSemaphore.Release(2);
}
Here is the Exception, took a few to get it for some odd reason.
System.InvalidOperationException was unhandled by user code
Message=An asynchronous call is already in progress. It must be completed or canceled before you can call this method.
Source=System
StackTrace:
at System.Net.Mail.SmtpClient.SendAsync(MailMessage message, Object userToken)
at DFW.Infrastructure.Communications.SmtpMail.SendMessage() in SmtpMail.cs:line 71
at EmaiProcessorService.EmailQueueService.b_0(EmailQueue message) in Service1.cs:line 57
at System.Threading.Tasks.Parallel.<>c_DisplayClass2d2.<ForEachWorker>b__23(Int32 i)
at System.Threading.Tasks.Parallel.<>c__DisplayClassf1.b__c()
InnerException:
Seems my waitone is getting obliterated by System.Threading.Tasks.Parallel

Okay, now that we've got the error text, it seems fairly clear:
Message=An asynchronous call is already in progress. It must be completed or canceled before you can call this method.
This concurs with the documentation:
Two simple options:
Create a fixed number of clients, and a queue of messages to send. Make each client take a message from the queue each time it finishes, until the queue is empty. BlockingCollection<T> is good for this.
Create a new SmtpClient per message. This could cause you to effectively launch a DOS attack on your SMTP server, which isn't ideal.
To be honest, it's not really clear why you're using SendAsync when you're then just waiting for the message to be sent anyway...

I'm not clear on why you're using a Semaphore here, but you're almost certainly using it incorrectly. You're creating a new semaphore instance for each call to SendMessage. Also, you're calling WaitOne on it once, and then calling Release(2), so eventually you'll have more releases than acquires. That's probably what causes your InvalidOperationException.
It doesn't do you any good to parallelize processing of the email queue, since you can only send one message at a time. And trying to do it asynchronously inside of the Parallel.Foreach is just more needless complication.
You're better off using something like ThreadPool.QueueUserWorkItem, and having a simple loop that sends one message at a time.
List<EmailQueue> emailQueue =
_repository.Select<EmailQueue>().Where(x => x.EmailStatuses.EmailStatus == "Pending").ToList();
ThreadPool.QueueUserWorkItem(ProcessEmailQueue, emailQueue);
void ProcessEmailQueue(object state)
{
List<EmailQueue> emailQueue = (List<EmailQueue>)state;
foreach (var message in EmailQueue)
{
// Format and send message here.
}
}
Alternatively, you can do the same thing with a Task. The point is that you just need a single thread to process the queue sequentially. Since you can't send more than one message at a time, Parallel.ForEach doesn't do you any good.
EDIT:
If you need to do multiple sends at a time, you can probably modify your original code. First, initialize the semaphore at class scope:
private static Semaphore mailSendSemaphore = new Semaphore(10, 10);
Then, in your SendMessage method:
bool SendMessage()
{
// acquire semaphore. This will block until there's a slot available.
mailSendSemaphore.WaitOne();
try
{
// do all your processing here, including sending the message.
// use Send rather than SendAsync
}
finally
{
mailSendSemaphore.Release();
}
}
There's no need to use SendAsync.