EasyNetQ with Windows Service Exception - c#

I have been running a simple Windows Service, with a EasyNetQ,
but when i try to publish a message I have been receiving an Exception.
Exception:
Pubisher confirms timed out after 10 seconds waiting for ACK or NACK from sequence
Here are my publish tests:
try {
var queue = _bus.Advanced.QueueDeclare("api-request-history");
_bus.Send(queue.Name, message);
} catch (Exception e) {
Logger.FatalException(string.Format("Message were not queued in queue: {0}", queue.Name), e);
OnErrorOccurred(e);
}
Here is my Subscription tests:
public override void Subscribe(Action<IReceiveRegistration> execution) {
var queue = _bus.Advanced.QueueDeclare("api-request-history");
try {
while (_bus.IsConnected) {
_bus.Receive(queue.Name, execution);
}
} catch (Exception e) {
Logger.ErrorException("Error occurred", e);
} finally {
if (_bus != null) {
_bus.Dispose();
}
}
}
Here is the Invokation of Subscribe:
_queueConsumer.Subscribe(x => x.Add<ApiRequestHistory>(message => {
Logger.Info("Initializing subscribtion sequence!");
Logger.Info("Message data is being saved...");
Execute();
Logger.Info("Message data saving is completed!");
Thread.Sleep(100);
wait.Set();
}));
wait.Wait();

I think you are subscribing the wrong way. You shouldn't be calling _bus.Receive(queueName) more than once for the same queue (from EasyNETQ docs):
Note: You probably do not want to call bus.Receive more than once for the same queue. This will create a new consumer on the queue and RabbitMQ will round-robin between them. If you are consuming different types on different Receive calls (and thus different consumers), some of your messages will end up on the error queue because EasyNetQ will not find a handler for your message type associated with the consumer on which it is consumed.
Just do it once and your consumers (per type) will be registered.
This is specially true as it seems you are using publisher confirms. I don't know your scenario fully well, but usually that is not needed unless you have a transactional RPC call model.
Check the docs on publisher confirms.

Related

C# How to force the wait for the connection to a WCF service

I have a service running as local SYSTEM that launches another application with the user credentials.
That second app is only a tray icon that shows balloon tips to the user with the string received using the callback method. This second application connects to the WCF in duplex mode.
My problem is that for some reason the connection to the WCF is finalized at the end of the method Main. So I cannot send a callback message to the app right after the execution, included in the last line "kiosk.MyStart(args);". there the callback is still pointing to null.
Any idea how could I solve this issue?
static void Main(string []args)
{
if (Environment.UserInteractive)
{
// Start the WCf service
var host = new ServiceHost(typeof(WcfService));
host.Open();
//Launch the Kiosk Agent which connects to the WCF
bool ret = ProcessAsUser.Launch("C:\\Program Files (x86)\\KIOSK\\KioskAgent.exe");
WinService kiosk = new WinService(args);
// some checks and a welcome message is sent to the user.
kiosk.MyStart(args);
//...
//...
}
}
Edit: to clarify a bit more, inside kiosk.MyStart method is where I try to execute the callback to show a welcome message, but the callback is still NULL.
As a result I assume that the client was not properly started for any reason and I launch it once again...
if (WcfService.Callback != null)
WcfService.Callback.UIMessageOnCallback(UIMessage);
else
ProcessAsUser.Launch("C:\\Program Files (x86)\\KIOSK\\KioskAgent.exe");
Add a try catch block over the callback method, if the client not reachable it falls in the catch you can unsubscribe it. Is also good practice send a keepalive message to your client, to check if it available.
private void InformClient(ClientInfo clientInfo)
{
var subscribers = this._subscriberRepository.GetAll();
foreach (var subscriber in subscribers)
{
try
{
if (subscriber.Callback.FireInformClient(clientInfo));
{
//If subscriber not reachable, unsubscribe it
this._subscriberRepository.Unsubscribe(subscriber.ClientId);
}
}
catch (Exception exception)
{
//If subscriber not reachable, unsubscribe it
this._subscriberRepository.Unsubscribe(subscriber.ClientId);
Log.Error(nameof(InformClient), exception);
}
}
}
IClientCallback
public interface IClientCallback
{
[OperationContract]
bool FireInformClient(ClientInfo clientInfo);
}
If you have more subscribers for example a terminal, server create a subscriberRepository to manage all subscribers.
var callback = OperationContext.Current.GetCallbackChannel<IClientCallback>();
if (this._subscriberRepository.Subscribe(clientId, callback))
{
return true;
}

WCF CommunicationException When Subscribed Clients Abort Abnormally

In a WCF publish/subscribe setup, I currently have an Unsubscribe() method in place to gracefully disconnect clients from the WCF host when the client is closed or needs to stop listening; however, this does not handle cases in which the client aborts forcefully or abnormally, such as the computer itself losing power. If a client application dies in such a way, then its channel remains and the following error is received at the publisher the next time it tries to send out messages:
ExceptionDetail> was caught
The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.
Clients subscribe anonymously, and the publisher follows a multicasting structure (any subscribed clients/channels should receive the message). Although I am able to catch the exception, I do not know how to single out the faulty channel from this point in the code in order to dispose of it and allow other clients to continue receiving messages. My publishing code looks similar to the following:
public static void Publish(DateTime sendTimeStamp, DataTable sendTable)
{
InstanceContext context = new InstanceContext(null, new PublishStatus());
MessagesClient publishingClient = new MessagesClient(context);
try {
publishingClient.PublishMessage(sendTimeStamp, sendTable);
if (publishingClient.State != CommunicationState.Faulted)
publishingClient.Close();
else
publishingClient.Abort();
}
catch (CommunicationException ex)
{
// This is where the error is caught
}
catch (TimeoutException ex)
{
publishingClient.Abort();
}
catch (Exception ex)
{
publishingClient.Abort();
throw ex;
}
}
Is it possible to isolate the faulty channel from this point (at which the exception first picks up on the issue) and dispose of it so that the publishing service itself can continue to send messages?
After some trial and error as well as exception research, an additional try-catch block in my WCF host was able to unsubscribe incorrectly aborted clients and keep the error from coming back to the publishing service. Posting a simple version here in case someone else stumbles on the same type of problem:
public static event MessageEventHandler MessageEvent;
public delegate void MessageEventHandler(object sender, ServiceEventArgs e);
IClientContract callback = null;
MessageEventHandler messageHandler = null;
public void Subscribe()
{
callback = OperationContext.Current.GetCallbackChannel<IClientContract>();
messageHandler = new MessageEventHandler(Publish_NewMessageEvent);
MessageEvent += messageHandler;
}
public void Unsubscribe()
{
MessageEvent -= messageHandler;
}
public void PublishMessage(DateTime timeStamp, DataTable table)
{
ServiceEventArgs se = new ServiceEventArgs();
se.timeStamp = timeStamp;
se.table = table;
MessageEvent(this, se);
}
public void Publish_NewMessageEvent(object sender, ServiceEventArgs e)
{
try
{
// This callback was causing the error, as the client would no longer exist but the channel would still be open and trying to receive the message
callback.ReceiveMessage(e.timeStamp, e.table);
}
catch
{
// Unsubscribe the dead client.
Unsubscribe();
}
}

What is better approach to listen in MultiThreading Service?

I am relatively new both to MSMQ and Threading in .NET. I have to create a service which listen in different threads, via TCP and SNMP, several network Devices and all this stuff run in dedicated threads, but here also is required to listen on MSMQ Queue from another applications.
I am analyzing another similar projects and there is used next logic:
private void MSMQRetrievalProc()
{
try
{
Message mes;
WaitHandle[] handles = new WaitHandle[1] { exitEvent };
while (!exitEvent.WaitOne(0, false))
{
try
{
mes = MyQueue.Receive(new TimeSpan(0, 0, 1));
HandleMessage(mes);
}
catch (MessageQueueException)
{
}
}
}
catch (Exception Ex)
{
//Handle Ex
}
}
MSMQRetrievalThread = new Thread(MSMQRetrievalProc);
MSMQRetrievalThread.Start();
But in another service (message dispatcher) I used asynchronous messages' reading based on MSDN Example:
public RootClass() //constructor of Main Class
{
MyQ = CreateQ(#".\Private$\MyQ"); //Get or create MSMQ Queue
// Add an event handler for the ReceiveCompleted event.
MyQ.ReceiveCompleted += new
ReceiveCompletedEventHandler(MsgReceiveCompleted);
// Begin the asynchronous receive operation.
MyQ.BeginReceive();
}
private void MsgReceiveCompleted(Object source, ReceiveCompletedEventArgs asyncResult)
{
try
{
// Connect to the queue.
MessageQueue mq = (MessageQueue)source;
// End the asynchronous Receive operation.
Message m = mq.EndReceive(asyncResult.AsyncResult);
// Process received message
// Restart the asynchronous Receive operation.
mq.BeginReceive();
}
catch (MessageQueueException Ex)
{
// Handle sources of MessageQueueException.
}
return;
}
Does asynchronous handling suppose that every message will be handled in other than main thread?
Could and need this (2nd) approach be put in separate thread?
Please advice better approach or some simple alternatives.
Messages arrival in Queue doesn't have some rule-defined behavior. It may be that for long time no nay message will arrive or in one second there my arrive many (up to 10 or even more) messages. Based on actions defined in some message it will need to delete/change some objects having running threads.
I highly recommend using WCF for MSMQ.
http://msdn.microsoft.com/en-us/library/ms789048.aspx
This allows you to both asynchronous handle the incoming calls using the WCF threading model which allows for throttling, capping, retries, etc...

Service not receiving messages after Message Queuing service restarted

We have a service that receives messages from n message queues. However, if the Message Queuing service is restarted, the message retrieval service stops receiving messages even after the Message Queuing service has restarted successfully.
I have tried to specifically catch the MessageQueueException that is thrown in the message retrieval service and invoke the queue's BeginReceive method again. However, in the 2 seconds or so that it takes the Message Queuing service to restart, I get about 1875 instances of the exception and then the service stops functioning when another MessageQueueException is thrown in our StartListening method.
Is there an elegant way to recover from a Message Queuing service restart?
private void OnReceiveCompleted(object sender, ReceiveCompletedEventArgs e)
{
MessageQueue queue = (MessageQueue)sender;
try
{
Message message = queue.EndReceive(e.AsyncResult);
this.StartListening(queue);
if (this.MessageReceived != null)
this.MessageReceived(this, new MessageReceivedEventArgs(message));
}
catch (MessageQueueException)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, StringResource.LogMessage_QueueManager_MessageQueueException, queue.MachineName, queue.QueueName, queue.Path));
this.StartListening(queue);
}
}
public void StartListening(MessageQueue queue)
{
queue.BeginReceive();
}
I need to deal with the infinite loop issue this causes and clean it up a bit but you get the idea.
When the MessageQueueException occurs, invoke the RecoverQueue method.
private void RecoverQueue(MessageQueue queue)
{
string queuePath = queue.Path;
bool queueRecovered = false;
while (!queueRecovered)
{
try
{
this.StopListening(queue);
queue.Close();
queue.Dispose();
Thread.Sleep(2000);
MessageQueue newQueue = this.CreateQueue(queuePath);
newQueue.ReceiveCompleted += new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
this.StartListening(newQueue);
LogUtility.LogInformation(String.Format(CultureInfo.InvariantCulture, "Message queue {0} recovered successfully.", newQueue.QueueName));
queueRecovered = true;
}
catch (Exception ex)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, "The following error occurred while trying to recover queue: {0} error: {1}", queue.QueueName, ex.Message));
}
}
}
public void StopListening(MessageQueue queue)
{
queue.ReceiveCompleted -= new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
}
Upon receiving the exception that is the result of the service restarting, you have to release the old MessageQueue, i.e. unwiring your ReceiveCompleted event, disposing the MessageQueue, etc. Then create a new instance of the MessageQueue and hook up to the ReceiveCompleted event again on the new MessageQueue instance.
Alternatively, you can use a polling method that creates a new instance on a certain interval, calls MessageQueue.Receive(TimeSpan), will wait for an incoming message or until the timeout occurs. In which case you handle the message and destroy the MessageQueue instance and start the iteration again.
By recreating the MessageQueue each time, you ensure a built in recovery. Also, the overhead of creating the MessageQueue is minimal due to internal caching of the underlying queue.
Pseudo-code...
while (!notDone)// or use a timer or periodic task of some sort...
{
try
{
using (MessageQueue queue = new MessageQueue(queuePath))
{
Message message = queue.Receive(TimeSpan.FromMilliseconds(500));
// process message
}
}
catch (MessageQueueException ex)
{
// handle exceptions
}
}

Invalid Operation Exception In Calling Async Method from inside a Parallel.ForEach Loop

I have inherited a windows service that processes a large number of e-mails in a queue. Sounds simple, Grab queue, send e-mail, if SmtpClient.SendAsync does not return an error from the call back then flag the e-mail in the DB as being sent.. I am using a Semaphore to waitone on the thread so multiple calls can be made to the Async Send method of the SMTP Client. This is the only way I can get the status and per Microsoft docs it has to finish the operation before another call can be made async. So now for the fun part. I decided to use a Parallel.ForEach to get he queue like so. This method is called in the Windows Service OnStart. Please note I have tried calling this method on a separate Thread and get the same results.
I am thinking that either A, I am missing something obvious, due to my lack of knowledge on threading, or something is flat bugged. Most likely A.
private static void ProcessEmailQueue()
{
List<EmailQueue> emailQueue =
_repository.Select<EmailQueue>().Where(x => x.EmailStatuses.EmailStatus == "Pending").ToList();
Parallel.ForEach(emailQueue, message =>
{
_smtpMail.FromAddress = message.FromAddress;
_smtpMail.ToAddress = message.ToAddress;
_smtpMail.Subject = message.Subject;
_smtpMail.SendAsHtml = message.IsHtml > 0;
_smtpMail.MessageBody = message.MessageBody;
_smtpMail.UserToken = message.EmailQueueID;
bool sendStatus = _smtpMail.SendMessage();
// THIS BLOWS UP with InvalidOperation Exception
});
}
Here is the SMTP Method being called from withing the loop.
public bool SendMessage()
{
mailSendSemaphore = new Semaphore(0, 10); // This is defined as private static Semaphore mailSendSemaphore;
try
{
var fromAddress = new MailAddress(FromAddress);
var toAddress = new MailAddress(ToAddress);
using (var mailMessage = new MailMessage(fromAddress, toAddress))
{
mailMessage.Subject = Subject;
mailMessage.IsBodyHtml = SendAsHtml;
mailMessage.Body = MessageBody;
Envelope = mailMessage;
smtp.SendCompleted += smtp_SendCompleted;
smtp.SendAsync(mailMessage, UserToken);
mailSendSemaphore.WaitOne();
return _mailSent;
}
}
catch (Exception exception)
{
_logger.Error(exception);
return _mailSent;
}
}
CALLBACK For Smtp Send
private void smtp_SendCompleted(object sender, AsyncCompletedEventArgs e)
{
if (e.Cancelled)
{
}
if (e.Error != null)
{
}
else
{
_mailSent = true;
}
mailSendSemaphore.Release(2);
}
Here is the Exception, took a few to get it for some odd reason.
System.InvalidOperationException was unhandled by user code
Message=An asynchronous call is already in progress. It must be completed or canceled before you can call this method.
Source=System
StackTrace:
at System.Net.Mail.SmtpClient.SendAsync(MailMessage message, Object userToken)
at DFW.Infrastructure.Communications.SmtpMail.SendMessage() in SmtpMail.cs:line 71
at EmaiProcessorService.EmailQueueService.b_0(EmailQueue message) in Service1.cs:line 57
at System.Threading.Tasks.Parallel.<>c_DisplayClass2d2.<ForEachWorker>b__23(Int32 i)
at System.Threading.Tasks.Parallel.<>c__DisplayClassf1.b__c()
InnerException:
Seems my waitone is getting obliterated by System.Threading.Tasks.Parallel
Okay, now that we've got the error text, it seems fairly clear:
Message=An asynchronous call is already in progress. It must be completed or canceled before you can call this method.
This concurs with the documentation:
Two simple options:
Create a fixed number of clients, and a queue of messages to send. Make each client take a message from the queue each time it finishes, until the queue is empty. BlockingCollection<T> is good for this.
Create a new SmtpClient per message. This could cause you to effectively launch a DOS attack on your SMTP server, which isn't ideal.
To be honest, it's not really clear why you're using SendAsync when you're then just waiting for the message to be sent anyway...
I'm not clear on why you're using a Semaphore here, but you're almost certainly using it incorrectly. You're creating a new semaphore instance for each call to SendMessage. Also, you're calling WaitOne on it once, and then calling Release(2), so eventually you'll have more releases than acquires. That's probably what causes your InvalidOperationException.
It doesn't do you any good to parallelize processing of the email queue, since you can only send one message at a time. And trying to do it asynchronously inside of the Parallel.Foreach is just more needless complication.
You're better off using something like ThreadPool.QueueUserWorkItem, and having a simple loop that sends one message at a time.
List<EmailQueue> emailQueue =
_repository.Select<EmailQueue>().Where(x => x.EmailStatuses.EmailStatus == "Pending").ToList();
ThreadPool.QueueUserWorkItem(ProcessEmailQueue, emailQueue);
void ProcessEmailQueue(object state)
{
List<EmailQueue> emailQueue = (List<EmailQueue>)state;
foreach (var message in EmailQueue)
{
// Format and send message here.
}
}
Alternatively, you can do the same thing with a Task. The point is that you just need a single thread to process the queue sequentially. Since you can't send more than one message at a time, Parallel.ForEach doesn't do you any good.
EDIT:
If you need to do multiple sends at a time, you can probably modify your original code. First, initialize the semaphore at class scope:
private static Semaphore mailSendSemaphore = new Semaphore(10, 10);
Then, in your SendMessage method:
bool SendMessage()
{
// acquire semaphore. This will block until there's a slot available.
mailSendSemaphore.WaitOne();
try
{
// do all your processing here, including sending the message.
// use Send rather than SendAsync
}
finally
{
mailSendSemaphore.Release();
}
}
There's no need to use SendAsync.

Categories

Resources