In a WCF publish/subscribe setup, I currently have an Unsubscribe() method in place to gracefully disconnect clients from the WCF host when the client is closed or needs to stop listening; however, this does not handle cases in which the client aborts forcefully or abnormally, such as the computer itself losing power. If a client application dies in such a way, then its channel remains and the following error is received at the publisher the next time it tries to send out messages:
ExceptionDetail> was caught
The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it has been Aborted.
Clients subscribe anonymously, and the publisher follows a multicasting structure (any subscribed clients/channels should receive the message). Although I am able to catch the exception, I do not know how to single out the faulty channel from this point in the code in order to dispose of it and allow other clients to continue receiving messages. My publishing code looks similar to the following:
public static void Publish(DateTime sendTimeStamp, DataTable sendTable)
{
InstanceContext context = new InstanceContext(null, new PublishStatus());
MessagesClient publishingClient = new MessagesClient(context);
try {
publishingClient.PublishMessage(sendTimeStamp, sendTable);
if (publishingClient.State != CommunicationState.Faulted)
publishingClient.Close();
else
publishingClient.Abort();
}
catch (CommunicationException ex)
{
// This is where the error is caught
}
catch (TimeoutException ex)
{
publishingClient.Abort();
}
catch (Exception ex)
{
publishingClient.Abort();
throw ex;
}
}
Is it possible to isolate the faulty channel from this point (at which the exception first picks up on the issue) and dispose of it so that the publishing service itself can continue to send messages?
After some trial and error as well as exception research, an additional try-catch block in my WCF host was able to unsubscribe incorrectly aborted clients and keep the error from coming back to the publishing service. Posting a simple version here in case someone else stumbles on the same type of problem:
public static event MessageEventHandler MessageEvent;
public delegate void MessageEventHandler(object sender, ServiceEventArgs e);
IClientContract callback = null;
MessageEventHandler messageHandler = null;
public void Subscribe()
{
callback = OperationContext.Current.GetCallbackChannel<IClientContract>();
messageHandler = new MessageEventHandler(Publish_NewMessageEvent);
MessageEvent += messageHandler;
}
public void Unsubscribe()
{
MessageEvent -= messageHandler;
}
public void PublishMessage(DateTime timeStamp, DataTable table)
{
ServiceEventArgs se = new ServiceEventArgs();
se.timeStamp = timeStamp;
se.table = table;
MessageEvent(this, se);
}
public void Publish_NewMessageEvent(object sender, ServiceEventArgs e)
{
try
{
// This callback was causing the error, as the client would no longer exist but the channel would still be open and trying to receive the message
callback.ReceiveMessage(e.timeStamp, e.table);
}
catch
{
// Unsubscribe the dead client.
Unsubscribe();
}
}
Related
I am new to Azure Service Bus and appreciate any help I can get.
In my current project, using c# and Azure.Messaging.ServiceBus we use an On-Premise server running a "Task Engine" windows service that listens to various queues (including MSMQ) to receive and process messages. We are migrating to Azure Service Bus Queue now.
I implemented ReceiveMessageAsync() method to read and process messages. The connection is persistent because the base class of the Task Engine service is already running in loop. While the below code works fine from my local pc (connected to VPN), it fails with the following error as soon as it's deployed to the on-premise server. The server also uses up all memory and shuts down causing other queues to terminate.
Error messages:
Azure.Messaging.ServiceBus.ServiceBusException: Creation of ReceivingAmqpLink did not complete in 30000 milliseconds. (ServiceTimeout)
Azure.Messaging.ServiceBus.ServiceBusException: 'receiver31' is closed (GeneralError)
Note:
Private Endpoint is enabled on Azure Service Bus and we use token and client credentials to connect to Azure.
All code below works fine locally when run for more than 2 hours and processes messages as soon as they are manually sent to queue using Azure Portal.
Code:
public **override **void StartUp(ContextBase context)
{
// Save the thread context
base.StartUp(context);
//Get values from Config
_tenantId = System.Configuration.ConfigurationManager.AppSettings["tenant-id"];
_clientId = System.Configuration.ConfigurationManager.AppSettings["client-id"];
_clientSecret = System.Configuration.ConfigurationManager.AppSettings["client-secret"];
_servicebusNamespace = System.Configuration.ConfigurationManager.AppSettings["servicebus-namespace"];
_messageQueueName = System.Configuration.ConfigurationManager.AppSettings["servicebus-inbound-queue"];
getAzureServiceBusAccess();
// Set the running flag
_isRunning = true;
}
//Called when service is initialized and then when Reset Connection happens due to error
private static void getAzureServiceBusAccess()
{
var _token = new ClientSecretCredential(_tenantId, _clientId, _clientSecret);
var clientOptions = new ServiceBusClientOptions()
{
TransportType = ServiceBusTransportType.AmqpWebSockets
};
_serviceBusClient = new ServiceBusClient(_servicebusNamespace, _token, clientOptions);
_serviceBusReceiver = _serviceBusClient.CreateReceiver(_messageQueueName, new ServiceBusReceiverOptions());
}
public **override **void DoAction()
{
// Make sure we haven't shut down
if (_isRunning)
{
// Wait next message
tryReceiveMessages();
}
}
private async void tryReceiveMessages()
{
try
{
ServiceBusReceivedMessage message = null;
message = await _serviceBusReceiver.ReceiveMessageAsync();
if (message != null && _isRunning)
{
try
{
string _messageBody = message.Body.ToString();
// <<Send message body to Task Adapter that adds it to the database and processes the job>>
await _serviceBusReceiver.CompleteMessageAsync(message);
}
catch (ServiceBusException s)
{
Tracer.RaiseError(Source.AzureSB, "Azure Service Bus Queue resulted in exception when processing message.", s);
throw;
}
catch (Exception ex)
{
Tracer.RaiseError(Source.AzureSB, "Unexpected error occurred moving task from Azure Service Bus to database; attempting to re-queue message.", ex);
if (message != null)
await _serviceBusReceiver.AbandonMessageAsync(message);
}
}
}
catch (ServiceBusException s)
{
tryResetConnections(s);
}
catch(Exception ex)
{
Tracer.RaiseError(Source.AzureSB, "Azure Service Bus Queue reset connection error.", ex);
throw;
}
}
private void tryResetConnections(Exception exception)
{
try
{
if (DateTime.Now.Subtract(LastQueueReset).TotalSeconds > 1800)
{
LastQueueReset = DateTime.Now;
getAzureServiceBusAccess();
}
else
{
//Send notification email to dev group
}
}
catch (Exception ex)
{
throw;
}
}
private async void closeAndDisposeConnectionAsync()
{
try
{
await _serviceBusReceiver.DisposeAsync();
}
catch(Exception ex)
{
//Do not throw and eat exception - Receiver may have been already disposed
}
try
{
await _serviceBusClient.DisposeAsync();
}
catch (Exception ex)
{
//Do not throw and eat exception - Client may have been already disposed
}
}
We tried to open the network settings on Azure Service Bus to public but that didn't resolve the issue.
I have requested the DevOps team to open ports 443, 5671 and 5672 for AMQPWebSockets and still waiting to hear back to test.
I have installed a Windows Service for my project, this error pop out when I start the service.
error 1503 the service didn't respond to the start or control request in a timely fashion
However, this project works fine when I debug in Visual Studio Code but when I use Visual Studio 2017 to create and start the service by following this tutorial and
Few solution I tried but the error still the same. Here are the solution I have tried.
Use CCCleaner to scan and fix issues
Modify ServicesPipeTimeout to 180000
The Logservice below can write the String into a text file, where I analyze the service run up until which part then fails. The service only able to run until LogService("3"); then it fails at receive the bytes from port.
Here is the code:
public Service1()
{
InitializeComponent();
LogService("server");
try
{
LogService("1");
IPEndPoint anyIP = new IPEndPoint(IPAddress.Any, 0);
UdpClient udpListener = new UdpClient(514);
byte[] bReceive; string sReceive; string sourceIP;
Console.WriteLine("Waiting...");
/* Main Loop */
/* Listen for incoming data on udp port 514 (default for SysLog events) */
while (true)
{
LogService("2");
try
{
LogService("3");
bReceive = udpListener.Receive(ref anyIP);
LogService("4");
/* Convert incoming data from bytes to ASCII */
sReceive = Encoding.ASCII.GetString(bReceive);
LogService(sReceive);
/* Get the IP of the device sending the syslog */
sourceIP = anyIP.Address.ToString();
LogService("5");
LogService(sourceIP);
new Thread(new logHandler(sourceIP, sReceive).handleLog).Start();
/* Start a new thread to handle received syslog event */
LogService(sReceive);
}
catch (Exception ex) { Console.WriteLine(ex.ToString()); }
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
throw ex;
}
}
I'm not sure if the error is occur because of the codes or other reasons
Update
Refer to this post, I tried to turn off the firewall to receive all the connection, but the error still remain the same.
There was once my project successfully listen the data from the server after I add udpListener.Client.Bind(anyIP); into the code, but then after some modification it is not functioning again. I'm not sure is the Bind() make the code works even just for once.
This is the way I solve my error refer to this post. Anyway, I'm not completely understand the process behind this, if anyone have some good example or link in explaining this solution please don't hesitate to comment below.
protected override void OnStart(string[] args)
{
LogService("Service started");
NewThread = new Thread(runSysLog);
NewThread.Start();
}
protected override void OnStop()
{
LogService("Service stopped");
StopRequest.Set();
//NewThread.Join();
}
public void runSysLog()
{
try
{
AutoResetEvent StopRequest = new AutoResetEvent(false);
/* Main Loop */
while (true)
{
if (StopRequest.WaitOne(5000)) return;
try
{
//while (udpListener.Available > 0)
if (udpListener.Available > 0)
{
//Some code here
}
}
catch (Exception ex)
{
LogService("Whileloop exception: " +ex.ToString());
throw ex;
}
}
}
catch (Exception ex)
{
LogService("Run Sys Log Exception: " +ex.ToString());
throw ex;
}
}
I have a service running as local SYSTEM that launches another application with the user credentials.
That second app is only a tray icon that shows balloon tips to the user with the string received using the callback method. This second application connects to the WCF in duplex mode.
My problem is that for some reason the connection to the WCF is finalized at the end of the method Main. So I cannot send a callback message to the app right after the execution, included in the last line "kiosk.MyStart(args);". there the callback is still pointing to null.
Any idea how could I solve this issue?
static void Main(string []args)
{
if (Environment.UserInteractive)
{
// Start the WCf service
var host = new ServiceHost(typeof(WcfService));
host.Open();
//Launch the Kiosk Agent which connects to the WCF
bool ret = ProcessAsUser.Launch("C:\\Program Files (x86)\\KIOSK\\KioskAgent.exe");
WinService kiosk = new WinService(args);
// some checks and a welcome message is sent to the user.
kiosk.MyStart(args);
//...
//...
}
}
Edit: to clarify a bit more, inside kiosk.MyStart method is where I try to execute the callback to show a welcome message, but the callback is still NULL.
As a result I assume that the client was not properly started for any reason and I launch it once again...
if (WcfService.Callback != null)
WcfService.Callback.UIMessageOnCallback(UIMessage);
else
ProcessAsUser.Launch("C:\\Program Files (x86)\\KIOSK\\KioskAgent.exe");
Add a try catch block over the callback method, if the client not reachable it falls in the catch you can unsubscribe it. Is also good practice send a keepalive message to your client, to check if it available.
private void InformClient(ClientInfo clientInfo)
{
var subscribers = this._subscriberRepository.GetAll();
foreach (var subscriber in subscribers)
{
try
{
if (subscriber.Callback.FireInformClient(clientInfo));
{
//If subscriber not reachable, unsubscribe it
this._subscriberRepository.Unsubscribe(subscriber.ClientId);
}
}
catch (Exception exception)
{
//If subscriber not reachable, unsubscribe it
this._subscriberRepository.Unsubscribe(subscriber.ClientId);
Log.Error(nameof(InformClient), exception);
}
}
}
IClientCallback
public interface IClientCallback
{
[OperationContract]
bool FireInformClient(ClientInfo clientInfo);
}
If you have more subscribers for example a terminal, server create a subscriberRepository to manage all subscribers.
var callback = OperationContext.Current.GetCallbackChannel<IClientCallback>();
if (this._subscriberRepository.Subscribe(clientId, callback))
{
return true;
}
I have been running a simple Windows Service, with a EasyNetQ,
but when i try to publish a message I have been receiving an Exception.
Exception:
Pubisher confirms timed out after 10 seconds waiting for ACK or NACK from sequence
Here are my publish tests:
try {
var queue = _bus.Advanced.QueueDeclare("api-request-history");
_bus.Send(queue.Name, message);
} catch (Exception e) {
Logger.FatalException(string.Format("Message were not queued in queue: {0}", queue.Name), e);
OnErrorOccurred(e);
}
Here is my Subscription tests:
public override void Subscribe(Action<IReceiveRegistration> execution) {
var queue = _bus.Advanced.QueueDeclare("api-request-history");
try {
while (_bus.IsConnected) {
_bus.Receive(queue.Name, execution);
}
} catch (Exception e) {
Logger.ErrorException("Error occurred", e);
} finally {
if (_bus != null) {
_bus.Dispose();
}
}
}
Here is the Invokation of Subscribe:
_queueConsumer.Subscribe(x => x.Add<ApiRequestHistory>(message => {
Logger.Info("Initializing subscribtion sequence!");
Logger.Info("Message data is being saved...");
Execute();
Logger.Info("Message data saving is completed!");
Thread.Sleep(100);
wait.Set();
}));
wait.Wait();
I think you are subscribing the wrong way. You shouldn't be calling _bus.Receive(queueName) more than once for the same queue (from EasyNETQ docs):
Note: You probably do not want to call bus.Receive more than once for the same queue. This will create a new consumer on the queue and RabbitMQ will round-robin between them. If you are consuming different types on different Receive calls (and thus different consumers), some of your messages will end up on the error queue because EasyNetQ will not find a handler for your message type associated with the consumer on which it is consumed.
Just do it once and your consumers (per type) will be registered.
This is specially true as it seems you are using publisher confirms. I don't know your scenario fully well, but usually that is not needed unless you have a transactional RPC call model.
Check the docs on publisher confirms.
We have a service that receives messages from n message queues. However, if the Message Queuing service is restarted, the message retrieval service stops receiving messages even after the Message Queuing service has restarted successfully.
I have tried to specifically catch the MessageQueueException that is thrown in the message retrieval service and invoke the queue's BeginReceive method again. However, in the 2 seconds or so that it takes the Message Queuing service to restart, I get about 1875 instances of the exception and then the service stops functioning when another MessageQueueException is thrown in our StartListening method.
Is there an elegant way to recover from a Message Queuing service restart?
private void OnReceiveCompleted(object sender, ReceiveCompletedEventArgs e)
{
MessageQueue queue = (MessageQueue)sender;
try
{
Message message = queue.EndReceive(e.AsyncResult);
this.StartListening(queue);
if (this.MessageReceived != null)
this.MessageReceived(this, new MessageReceivedEventArgs(message));
}
catch (MessageQueueException)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, StringResource.LogMessage_QueueManager_MessageQueueException, queue.MachineName, queue.QueueName, queue.Path));
this.StartListening(queue);
}
}
public void StartListening(MessageQueue queue)
{
queue.BeginReceive();
}
I need to deal with the infinite loop issue this causes and clean it up a bit but you get the idea.
When the MessageQueueException occurs, invoke the RecoverQueue method.
private void RecoverQueue(MessageQueue queue)
{
string queuePath = queue.Path;
bool queueRecovered = false;
while (!queueRecovered)
{
try
{
this.StopListening(queue);
queue.Close();
queue.Dispose();
Thread.Sleep(2000);
MessageQueue newQueue = this.CreateQueue(queuePath);
newQueue.ReceiveCompleted += new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
this.StartListening(newQueue);
LogUtility.LogInformation(String.Format(CultureInfo.InvariantCulture, "Message queue {0} recovered successfully.", newQueue.QueueName));
queueRecovered = true;
}
catch (Exception ex)
{
LogUtility.LogError(String.Format(CultureInfo.InvariantCulture, "The following error occurred while trying to recover queue: {0} error: {1}", queue.QueueName, ex.Message));
}
}
}
public void StopListening(MessageQueue queue)
{
queue.ReceiveCompleted -= new ReceiveCompletedEventHandler(this.OnReceiveCompleted);
}
Upon receiving the exception that is the result of the service restarting, you have to release the old MessageQueue, i.e. unwiring your ReceiveCompleted event, disposing the MessageQueue, etc. Then create a new instance of the MessageQueue and hook up to the ReceiveCompleted event again on the new MessageQueue instance.
Alternatively, you can use a polling method that creates a new instance on a certain interval, calls MessageQueue.Receive(TimeSpan), will wait for an incoming message or until the timeout occurs. In which case you handle the message and destroy the MessageQueue instance and start the iteration again.
By recreating the MessageQueue each time, you ensure a built in recovery. Also, the overhead of creating the MessageQueue is minimal due to internal caching of the underlying queue.
Pseudo-code...
while (!notDone)// or use a timer or periodic task of some sort...
{
try
{
using (MessageQueue queue = new MessageQueue(queuePath))
{
Message message = queue.Receive(TimeSpan.FromMilliseconds(500));
// process message
}
}
catch (MessageQueueException ex)
{
// handle exceptions
}
}