We have a cloud service using a worker role to process messages it receives from a Topic set up on Azure Service Bus.
The message itself seems to arrive intact and is usually received and processed correctly. In some instances however, the message seems to stop processing (Logging abruptly ends and no more references to the message being processed are seen in our WadLogsTable). From my research, this might be happening due to the worker role keeping its connection open and idle for longer than seconds. How would I go about preventing these long-to-process messages from being abandoned?
The code for our worker role is below.
public class WorkerRole : RoleEntryPoint
{
private static StandardKernel _kernel;
private readonly ManualResetEvent _completedEvent = new ManualResetEvent(false);
private BaseRepository<CallData> _callDataRepository;
private BaseRepository<CallLog> _callLogRepository;
private SubscriptionClient _client;
private NamespaceManager _nManager;
private OnMessageOptions _options;
private BaseRepository<Site> _siteRepository;
public override void Run()
{
try
{
List<CallInformation> callInfo;
Trace.WriteLine("Starting processing of messages");
// Initiates the message pump and callback is invoked for each message that is received, calling close on the client will stop the pump.
_client.OnMessage(message =>
{
// Process message from subscription.
Trace.TraceInformation("Call Received. Ready to process message ");
message.RenewLock();
callInfo = message.GetBody<List<CallInformation>>();
writeCallData(callInfo);
Trace.TraceInformation("Call Processed. Clearing from topic.");
}, _options);
}
catch (Exception e)
{
Trace.TraceInformation("Error: " + e.Message + "---" + e.StackTrace);
}
}
private void writeCallData(List<CallInformation> callList)
{
try
{
Trace.TraceInformation("Calls received: " + callList.Count);
foreach (var callInfo in callList)
{
Trace.TraceInformation("Unwrapping call...");
var call = callInfo.CallLog.Unwrap();
Trace.TraceInformation("Begin Processing: Local Call " + call.ID + " with " + callInfo.DataPoints.Length + " datapoints");
Trace.TraceInformation("Inserting Call...");
_callLogRepository.ExecuteSqlCommand(/*SNIP: Insert call*/);
Trace.TraceInformation("Call entry written. Now building datapoint list...");
var datapoints = callInfo.DataPoints.Select(datapoint => datapoint.Unwrap()).ToList();
Trace.TraceInformation("datapoint list constructed. Processing datapoints...");
foreach (var data in datapoints)
{
/*SNIP: Long running code. Insert our datapoints one at a time. Sometimes our messages die in the middle of this foreach. */
}
Trace.TraceInformation("All datapoints written for call with dependable ID " + call.Call_ID);
Trace.TraceInformation("Call Processed successfully.");
}
}
catch (Exception e)
{
Trace.TraceInformation("Call Processing Failed. " + e.Message);
}
}
public override bool OnStart()
{
try
{
var connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
_nManager = NamespaceManager.CreateFromConnectionString(connectionString);
_nManager.Settings.OperationTimeout = new TimeSpan(0,0,10,0);
var topic = new TopicDescription("MyTopic")
{
DuplicateDetectionHistoryTimeWindow = new TimeSpan(0, 0, 10, 0),
DefaultMessageTimeToLive = new TimeSpan(0, 0, 10, 0),
RequiresDuplicateDetection = true,
};
if (!_nManager.TopicExists("MyTopic"))
{
_nManager.CreateTopic(topic);
}
if (!_nManager.SubscriptionExists("MyTopic", "AllMessages"))
{
_nManager.CreateSubscription("MyTopic", "AllMessages");
}
_client = SubscriptionClient.CreateFromConnectionString(connectionString, "MyTopic", "AllMessages",
ReceiveMode.ReceiveAndDelete);
_options = new OnMessageOptions
{
AutoRenewTimeout = TimeSpan.FromMinutes(5),
};
_options.ExceptionReceived += LogErrors;
CreateKernel();
_callLogRepository.ExecuteSqlCommand(/*SNIP: Background processing*/);
}
catch (Exception e)
{
Trace.TraceInformation("Error on roleStart:" + e.Message + "---" + e.StackTrace);
}
return base.OnStart();
}
public override void OnStop()
{
// Close the connection to Service Bus Queue
_client.Close();
_completedEvent.Set();
}
void LogErrors(object sender, ExceptionReceivedEventArgs e)
{
if (e.Exception != null)
{
Trace.TraceInformation("Error: " + e.Exception.Message + "---" + e.Exception.StackTrace);
_client.Close();
}
}
public IKernel CreateKernel()
{
_kernel = new StandardKernel();
/*SNIP: Bind NInjectable repositories */
return _kernel;
}
}
Your Run method does not go on indefinitely. It should look like this:
public override void Run()
{
try
{
Trace.WriteLine("WorkerRole entrypoint called", "Information");
while (true)
{
// Add code here that runs in the role instance
}
}
catch (Exception e)
{
Trace.WriteLine("Exception during Run: " + e.ToString());
// Take other action as needed.
}
}
Taken from the docs:
The Run is considered the Main method for your application. Overriding
the Run method is not required; the default implementation never
returns. If you do override the Run method, your code should block
indefinitely. If the Run method returns, the role is automatically
recycled by raising the Stopping event and calling the OnStop method
so that your shutdown sequences may be executed before the role is
taken offline.
TheDude's response is very close to the correct answer! It turns out he's right that the run method needs to stay alive instead of returning immediately. With Azure Service Bus's message pump mechanism though, you can't place the _client.onMessage(...) inside a while loop, as this results in an error (The message pump has already been initialized).
What actually needs to happen is a a manual reset event needs to be created before the worker role begins executing, and then waited after the message pump code is executed. For documentation on ManualResetEvent, see https://msdn.microsoft.com/en-us/library/system.threading.manualresetevent(v=vs.110).aspx. Additionally, the process is described here: http://www.acousticguitar.pro/questions/607359/using-queueclient-onmessage-in-an-azure-worker-role
My final worker role class looks like this:
public class WorkerRole : RoleEntryPoint
{
private static StandardKernel _kernel;
private readonly ManualResetEvent _completedEvent = new ManualResetEvent(false);
private BaseRepository<CallLog> _callLogRepository;
private SubscriptionClient _client;
private MessagingFactory _mFact;
private NamespaceManager _nManager;
private OnMessageOptions _options;
public override void Run()
{
ManualResetEvent CompletedEvent = new ManualResetEvent(false);
try
{
CallInformation callInfo;
// Initiates the message pump and callback is invoked for each message that is received, calling close on the client will stop the pump.
_client.OnMessage(message =>
{
// Process message from subscription.
Trace.TraceInformation("Call Received. Ready to process message " + message.MessageId);
callInfo = message.GetBody<CallInformation>();
WriteCallData(callInfo);
Trace.TraceInformation("Call Processed. Clearing from topic.");
}, _options);
}
catch (Exception e)
{
Trace.TraceInformation("Error: " + e.Message + "---" + e.StackTrace);
}
CompletedEvent.WaitOne();
}
private void writeCallData(List<CallInformation> callList)
{
try
{
Trace.TraceInformation("Calls received: " + callList.Count);
foreach (var callInfo in callList)
{
Trace.TraceInformation("Unwrapping call...");
var call = callInfo.CallLog.Unwrap();
Trace.TraceInformation("Begin Processing: Local Call " + call.ID + " with " + callInfo.DataPoints.Length + " datapoints");
Trace.TraceInformation("Inserting Call...");
_callLogRepository.ExecuteSqlCommand(/*SNIP: Insert call*/);
Trace.TraceInformation("Call entry written. Now building datapoint list...");
var datapoints = callInfo.DataPoints.Select(datapoint => datapoint.Unwrap()).ToList();
Trace.TraceInformation("datapoint list constructed. Processing datapoints...");
foreach (var data in datapoints)
{
/*SNIP: Long running code. Insert our datapoints one at a time. Sometimes our messages die in the middle of this foreach. */
}
Trace.TraceInformation("All datapoints written for call with dependable ID " + call.Call_ID);
Trace.TraceInformation("Call Processed successfully.");
}
}
catch (Exception e)
{
Trace.TraceInformation("Call Processing Failed. " + e.Message);
}
}
public override bool OnStart()
{
try
{
var connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
_nManager = NamespaceManager.CreateFromConnectionString(connectionString);
_nManager.Settings.OperationTimeout = new TimeSpan(0,0,10,0);
var topic = new TopicDescription("MyTopic")
{
DuplicateDetectionHistoryTimeWindow = new TimeSpan(0, 0, 10, 0),
DefaultMessageTimeToLive = new TimeSpan(0, 0, 10, 0),
RequiresDuplicateDetection = true,
};
if (!_nManager.TopicExists("MyTopic"))
{
_nManager.CreateTopic(topic);
}
if (!_nManager.SubscriptionExists("MyTopic", "AllMessages"))
{
_nManager.CreateSubscription("MyTopic", "AllMessages");
}
_client = SubscriptionClient.CreateFromConnectionString(connectionString, "MyTopic", "AllMessages",
ReceiveMode.ReceiveAndDelete);
_options = new OnMessageOptions
{
AutoRenewTimeout = TimeSpan.FromMinutes(5),
};
_options.ExceptionReceived += LogErrors;
CreateKernel();
_callLogRepository.ExecuteSqlCommand(/*SNIP: Background processing*/);
}
catch (Exception e)
{
Trace.TraceInformation("Error on roleStart:" + e.Message + "---" + e.StackTrace);
}
return base.OnStart();
}
public override void OnStop()
{
// Close the connection to Service Bus Queue
_client.Close();
_completedEvent.Set();
}
void LogErrors(object sender, ExceptionReceivedEventArgs e)
{
if (e.Exception != null)
{
Trace.TraceInformation("Error: " + e.Exception.Message + "---" + e.Exception.StackTrace);
_client.Close();
}
}
public IKernel CreateKernel()
{
_kernel = new StandardKernel();
/*SNIP: Bind NInjectable repositories */
return _kernel;
}
}
You'll notice the presence of the ManualResetEvent and the invocation of WaitOne() at the end of my Run method. I hope someone finds this helpful!
Related
The problem is that dispatcher blocks the MainWindow thread until it is executed all of the queued AddOutputLine()s. Can these calls be non-blocking for a wpf control?
//TaskPage.xaml.cs
[MTAThread]
public void AddOutputLine(string line = "")
{
try {
Session.TaskPage.Dispatcher.BeginInvoke(new Action(delegate
{
txtOutput.Text += "[" + DateTime.Now + "] " + line + Environment.NewLine;
txtOutput.ScrollToEnd();
}));
} catch (Exception ex) {
Program.SetCurrentStatus("Error occured : "+ex.Message);
}
}
In this class that invokes the Action parameter:
//BackgroundUploader.cs
private Action<string> _log;
public BackgroundUploader(Client client, Action<string> log = null)
{
_log = log;
_client = client;
}
while doing this:
//BackgroundUploader.cs
public void RunThreadProc()
{
try
{
_log?.Invoke("Timeout interval set for BackGround Task.");
while (!Stop)
{
//the problematic blocking of WPF Control starts here
var inOutFiles = CreateXML(_sqlStrings, _outputNames, _outputDirectory);
Send(inOutFiles, _outputDirectory);
LastRunTime = DateTime.Now;
_log?.Invoke($"Waiting for {_waitMinutes} minutes...");
//the Control returns to processing messages
System.Threading.Thread.Sleep((int)TimeSpan.FromMinutes(_waitMinutes).TotalMilliseconds);
}
_log?.Invoke("BackGroundTask canceled.");
}
catch (Exception ex)
{
_log?.Invoke($"Error while executing background task: {ex.Message}");
}
}
The RunThreadProc() is called from Control like this:
//TaskPage.xaml.cs
_backgroundThread = new System.Threading.Thread(bU.RunThreadProc);
_backGroundThread.Start();
I also cannot use Bcl.Async because it's failing on Task result with method that uses async modifier on Windows Server 2003 target OS.
edit: I encountered the problem inside of the background class, its methods run synchronously, even though they are in a different thread.
I have an async function which still freezes / lags the UI thread for me when I execute it. This is my function calling it.
private void TcpListenerLogic(object sender, string e)
{
Application.Current.Dispatcher.BeginInvoke((Action)async delegate {
try
{
dynamic results = JsonConvert.DeserializeObject<dynamic>(e);
if (results.test_id != null)
{
// Get properties for new anchor
string testInformation = await CommunicationCommands.getJsonFromURL(
"http://" + ServerIP + ":" + ServerPort + "/api/" + results.test_id);
}
}
catch (Exception exception)
{
// Writing some Trace.WriteLine()'s
}
});
}
And this is the async function that freezes my UI Thread
public static async Task<string> getJsonFromURL(string url)
{
try
{
string returnString = null;
using (System.Net.WebClient client = new System.Net.WebClient())
{
returnString = await client.DownloadStringTaskAsync(url);
}
return returnString;
}
catch (Exception ex)
{
Debug.WriteLine(ex.ToString());
return null;
}
}
I already tried to make everything in TcpListenerLogic run in a new Thread:
new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
}).Start();
Which resulted in the whole UI completely freezing. And I tried to make TcpListenerLogic async and await the dispatcher, which also made everything freeze permanently. I also tried to make TcpListenerLogic async and leave the dispatcher. The dispatcher is only there because I normally have some UI code in there, which I left out for my tests.
I have ventured far through the internet, but no BackgroundWorker, ThreadPool or other methods helped me in my endeavour.
If anyone has help for this particular problem, or a resource that would improve my understanding of async functions in C#, I would much appreciate it.
Edit
As requested a deeper insight in how this event handler is called.
I have System.Net.Websocket, which is connected to the Backend API I am working with and triggers an event, everytime he receives new Data. To guarantee the socket listens as longs as it is open, there is a while loop which checks for the client state:
public event EventHandler<string> TcpReceived;
public async void StartListener(string ip, int port, string path)
{
try
{
using (client = new ClientWebSocket())
{
try
{ // Connect to backend
Uri serverUri = new Uri("ws://" + ip + ":" + port.ToString() + path );
await client.ConnectAsync(serverUri, CancellationToken.None);
}
catch (Exception ex)
{
BackendSettings.IsConnected = false;
Debug.WriteLine("Error connecting TCP Socket: " + ex.ToString());
}
state = client.State;
// Grab packages send in backend
while (client.State == WebSocketState.Open || client.State == WebSocketState.CloseSent)
{
try
{
// **Just formatting the received data until here and writing it into the "message" variable**//
TcpReceived(this, message);
// Close connection on command
if (result.MessageType == WebSocketMessageType.Close)
{
Debug.WriteLine("Closing TCP Socket.");
shouldstayclosed = true;
await client.CloseAsync(WebSocketCloseStatus.NormalClosure, string.Empty, CancellationToken.None);
break;
}
state = client.State;
}
catch
{
BackendSettings.IsConnected = false;
state = client.State;
}
}
state = client.State;
}
}
catch (Exception ex)
{
// Some error messages and settings handling
}
}
The Event has a handler attached:
TcpReceived += TcpListener_TcpReceived;
And this is the Handler, which calls the previously seen "TcpListenereLogic".
private void TcpListener_TcpReceived(object sender, string e)
{
TcpListenerLogic(sender, e);
//App.Current.Dispatcher.BeginInvoke(new Action(() => {
// TcpListenerLogic(sender, e);
//}));
//new Thread(() =>
//{
// Thread.CurrentThread.IsBackground = true;
// TcpListenerLogic(sender, e);
//}).Start();
}
I previously had the "TcpListenereLogic" as the handler, but I wanted to try different methods to call it. I also left in the commented out part, to show how the call of "TcpListenereLogic" looked already. All my attempts were with all mentioned setups and sadly lead to nothing.
Thank you very much #TheodorZoulias for helping me to find the solution to my problem.
It turns out it wasn't the async function itself, but rather how often it gets called. It got called roughly ~120 times every second.
My solution starts by calling the Listener method over a new Thread:
new Thread(() =>
{
Thread.CurrentThread.IsBackground = true;
MainWindow.tcpListener.StartListener(ip, portNumber, "/api/");
}).Start();
To limit the amount of calls that happen every second I added a dispatcher timer, that resets a bool after it has been used for a call, by my Event.
readonly System.Windows.Threading.DispatcherTimer packageIntervallTimer =
new System.Windows.Threading.DispatcherTimer();
bool readyForNewPackage = true;
private void ReadyForPackage(object sender, EventArgs e)
{
readyForNewPackage = true;
}
public async void StartListener(string ip, int port, string path)
{
packageIntervallTimer.Interval = TimeSpan.FromMilliseconds(50);
packageIntervallTimer.Tick += (s, e) => { Task.Run(() => ReadyForPackage(s, e)); };
packageIntervallTimer.Start();
Then I wrapped everything inside the while loop into an if condition based on the bool, the most important part was to have my "event EventHandler TcpReceived" in there:
// Grab packages sent in backend
while (client.State == WebSocketState.Open || client.State == WebSocketState.CloseSent)
{
if (readyForNewPackage == true)
{
readyForNewPackage = false;
try
{
....
TcpReceived(this, message);
....
}
catch
{
...
}
}
}
I added my TcpListenerLogic to the Eventhandler:
TcpReceived += TcpListenerLogic;
And my TcpListenerLogic now looked like this (names have been changed):
private async void TcpListenerLogic(object sender, string e)
{
try
{
dynamic results = JsonConvert.DeserializeObject<dynamic>(e);
if (results.test_id != null)
{
string testID = "";
if (results.test_id is JValue jValueTestId)
{
testID = jValueTestId.Value.ToString();
}
else if (results.test_id is string)
{
testID = results.test_id;
}
// Get properties for new object
string information = await CommunicationCommands.getJsonFromURL(
"http://" + ServerIP + ":" + ServerPort + "/api/" + testID );
if (information != null)
{
await App.Current.Dispatcher.BeginInvoke(new Action(() =>
{
// Create object out of the json string
TestStatus testStatus = new TestStatus();
testStatus.Deserialize(information);
if (CommunicationCommands.isNameAlreadyInCollection(testStatus.name) == false)
{
// Add new object to the list
CommunicationCommands.allFoundTests.Add(testStatus);
}
}));
{
}
catch (Exception exception)
{
....
}
}
Adding a new Thread to execute any step results in problems, so keep in mind that all this uses the thread created at the beginning for "StartListener"
I am trying to deploy my first Azure worker role and I have been running into this error when Run() method is called during service start up.
An unhandled exception of type 'System.StackOverflowException' occurred in Unknown Module.
I've tried to remote debug my code and the error is thrown at this line. MyPublisher is similar to MyQueue but it wraps Topic instead of a Queue. Any idea why QueueClient.OnMessage would cause StackOverflow?
Client.OnMessage(messageHandler, options);
Here is the partial code. My Apology if it is not formatted correctly (will try to format) or anything is missing in code.
public class MyQueue
{
String QueueName;
public QueueClient Client { get; protected set; }
public MyQueue(String queueName)
{
Trace.WriteLine($"Creating service Queue with name : {queueName} ");
QueueName = queueName;
}
public void EstableshConnection(string connectionString = null)
{
Trace.WriteLine($"Establishing connection with service Queue : {QueueName} ");
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
connectionString = connectionString ?? CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
NamespaceManager namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.QueueExists(QueueName))
namespaceManager.CreateQueue(QueueName);
Client = QueueClient.CreateFromConnectionString(connectionString, QueueName);
}
public void Send(BrokeredMessage message)
{
Trace.WriteLine($"Sending brokered message to queue : {QueueName} ");
if (Client != null && !Client.IsClosed)
Client.Send(message);
}
public void OnMessage(Action<BrokeredMessage> messageHandler)
{
Trace.WriteLine($"OnMessage handler: Queue Name : {QueueName} ");
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = true; // Indicates if the message-pump should call complete on messages after the callback has completed processing.
options.MaxConcurrentCalls = 1; // Indicates the maximum number of concurrent calls to the callback the pump should initiate
options.ExceptionReceived += LogErrors; // Allows users to get notified of any errors encountered by the message pump
//=====================StackOverFlowException on Client.OnMessage======
if (Client != null && !Client.IsClosed)
Client.OnMessage(messageHandler, options); //This is where I get StackOverflowException Error.
}
private void LogErrors(object sender, ExceptionReceivedEventArgs e)
{
if (e.Exception != null)
Trace.WriteLine("Queue client processing error: " + e.Exception.Message);
}
public void Disconnect()
{
Trace.WriteLine($"closing queue {QueueName}");
Client.Close();
}
}
Here is my workerrole implementation.
public class MyWorkerRole : RoleEntryPoint
{
#region Variables
ManualResetEvent CompletedEvent = new ManualResetEvent(false);
MyQueue RequestQueue; //for Request
MyPublisher ResponseTopicClient; //ReponseTopic to notify Subscriber when processing is completed
Public MyWorkerRole()
{
RequestQueue = new MyQueue("JobRequestQueue");
ResponseTopicClient = new MyPublisher("JobCompletedTopic");
}
public override bool OnStart()
{
try
{
RequestQueue.EstableshConnection();
ResponseTopicClient.EstableshConnection();
}
catch (Exception ex)
{
Trace.TraceWarning($"Trace: starting service failed. Error {ex.Message} ");
}
return base.OnStart();
}
public override void OnStop()
{
try
{
RequestQueue.Disconnect();
ResponseTopicClient.Disconnect();
CompletedEvent.Set();
}
catch (Exception ex)
{
Trace.TraceWarning($"Trace: stopping service failed with error. {ex.Message} ");
}
base.OnStop();
}
public override void Run()
{
try
{
Trace.WriteLine("Trace: Starting Message Processing");
//var receivedMessage2 = RequestQueue.Client.Receive(new TimeSpan(hours: 0, minutes: 2, seconds: 0));
RequestQueue.OnMessage((receivedMessage) =>
{
try
{
Guid resultGuid = (Guid)receivedMessage.Properties["CorrelationGuid"];
Trace.TraceWarning($"Trace: processing message with GUID {resultGuid}");
var messageToSend = JobProcessor.ProcessRequest(receivedMessage);
if (messageToSend == null)
{
Trace.TraceError("Trace: > Broken message!");
receivedMessage.Abandon();
return;
}
ResponseTopicClient.Send(messageToSend);
receivedMessage.Complete();
}
catch (Exception ex)
{
Trace.TraceError("Trace: Processing exception: " + ex.Message + "\nStack Trace" + ex.StackTrace);
Logger.Error("Processing exception: " + ex.Message + "\nStack Trace" + ex.StackTrace);
}
});
CompletedEvent.WaitOne();
}
catch (Exception ex)
{
Trace.TraceError("Trace: Run exception: " + ex.Message + "\nStack Trace" + ex.StackTrace);
}
finally
{
CompletedEvent.Set();
}
}
}
When your worker starts, it calls the Run method and in your code, you have :
//var receivedMessage2 = RequestQueue.Client.Receive(new TimeSpan(hours: 0, minutes: 2, seconds: 0));
RequestQueue.OnMessage((receivedMessage) =>
So, The code doesn't wait for a new message because the first line is commented and it calls the OnMessage method which recursibly calls itself again and again till the StackOverflowException gets fired
In all cases, you need to change the implementation because StackOverflowException will happen anyway when a new message is received
Figured it out. So, the issue was not with the code. Following error was reported in WADWindowsEventLogsTable under my storage account.
Faulting application name: WaWorkerHost.exe, version: 2.7.1198.768, time stamp: 0x57159090
Faulting module name: Microsoft.IntelliTrace.Profiler.SC.dll, version: 15.0.27128.1, time stamp: 0x5a1e2eb9
Exception code: 0xc00000fd
Fault offset: 0x000000000008ae7b
Faulting process id: 0xcf4
Faulting application start time: 0x01d3b75ed89dc2f9
Faulting application path: F:\base\x64\WaWorkerHost.exe
Faulting module path: F:\plugins\IntelliTrace\Runtime\x64\Microsoft.IntelliTrace.Profiler.SC.dll
This gave me a hint about disabling the IntelliTrace and it worked just fine. Here is how you can disable while publishing a package through VS 2017.
1.) Right click on your worker role project and select publish from menu
2.) In Settings page->Advanced Settings, uncheck "Enable IntelliTrace" option.
I've seen a fair few posts regarding this issue but non that seem to fit my criteria.
Having said that, this is the first one that I've built that "needs" to perform a tremendous amount of logic. All works great for a couple days but after that logic stops working with no log in the event viewer nor firing of an email (exceptions).
I'm wondering if my logic is not correct... Looking for advice and pointers.
public partial class QuayService : ServiceBase
{
private System.Timers.Timer m_mainTimer;
private bool m_timerTaskSuccess;
private Email _email;
public QuayService()
{
InitializeComponent();
_email = new Email();
}
protected override void OnStart(string[] args)
{
try
{
// Create and start a timer.
m_mainTimer = new System.Timers.Timer();
m_mainTimer.Interval = 5000; // every 5 seconds
m_mainTimer.Elapsed += m_mainTimer_Elapsed;
m_mainTimer.AutoReset = false; // makes it fire only once
m_mainTimer.Start(); // Start
m_timerTaskSuccess = false;
_email.SendEmail("Quay", "(Shopify)Quay Service Started",
"(Shopify)Quay Service Successfuly Started");
EventLog.WriteEntry("(Shopify)Quay Service Started...");
}
catch (Exception ex)
{
// Log/Send Email
_email.SendEmail("Quay", "Error starting (Shopify)Quay Service", ex.Message + " " +
ex.InnerException.Message);
EventLog.WriteEntry("Error starting (Shopify)Quay service and timer..." + ex.Message + " " +
ex.InnerException.Message);
}
}
protected override void OnStop()
{
try
{
// Service stopped. Also stop the timer.
m_mainTimer.Stop();
m_mainTimer.Dispose();
m_mainTimer = null;
_email.SendEmail("Quay", "(Shopify)Quay Service stopped",
"(Shopify)Quay Service Successfuly Stopped");
EventLog.WriteEntry("(Shopify)Quay Service stopped...");
}
catch (Exception ex)
{
_email.SendEmail("Quay", "Error stopping (Shopify)Quay Service", ex.Message + " " +
ex.InnerException.Message);
// Log/Send Email
EventLog.WriteEntry("Error stopping (Shopify)Quay timer and service..." + ex.Message + " " +
ex.InnerException.Message);
}
}
void m_mainTimer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
try
{
var orderUoW = new OrderUoW();
orderUoW.Create();
m_timerTaskSuccess = true;
}
catch (Exception ex)
{
//Error with timer elapsed
m_timerTaskSuccess = false;
_email.SendEmail("Quay", "Error creating (Shopify)Quay order(s)", ex.Message + " " +
ex.InnerException.Message);
EventLog.WriteEntry("Error creating (Shopify)Quay order(Time elapsed event)..." + ex.Message
+ " " + ex.InnerException.Message);
}
finally
{
if (m_timerTaskSuccess)
{
m_mainTimer.Start();
}
}
}
}
To get to this point, I "googled" and "SO'ed" to find the best use of timer... I also have a naïve test layer that reproduces what's in the service, I can run through that fine without any exceptions but this one is driving me nuts.
Any help truly appreciated!
//EDIT
A little explanation on:
var orderUoW = new OrderUoW();
orderUoW.Create();
OrderUoW is a class in my service that inherits from BatchOrder which is responsible for many actions including connecting to two databases and polling of the shopify API... OrderUow is purely to decouple from the business layer but giving access to "x" methods.
All works flawlessly for two days but just seems to halt. I'm not hitting request limits within shopify... so at the moment, I'm at a loss.
After many moments of scratching my head and extreme annoyance with the MS Timer Bug.
Not only did it swallow exceptions, the timer did not raise the event! My service was hanging, which was particularly difficult because it also swallowed exceptions.
Now I did go down the route of half implementing with Threading.Timer
Don't use System.Windows.Forms.Timer because it won't work (this only makes sense).
Don't use System.Threading.Timer because it doesn't work, use System.Timers.Timer instead.
Don't use System.Timers.Timer because it doesn't work, use System.Threading.Timer instead.
Related question.
I really could not be bothered with the issues that came with these steps, as my aim was to focus on the business logic. So I made the decision to use Quartz.Net.
It made my life much easier and seems to work perfectly! 45-60 minutes worth of work.
Basic Setup:
1, Nuget > Quartz
2, New Folder(s) > QuartzComponents > Jobs and Schedule
3, Added OrderJob.cs and OrderJobSchedule.cs in respective folders
OrderJob.cs logic:
public sealed class OrderJob : IJob
{
public void Execute(IJobExecutionContext context)
{
var orderUoW = new OrderUoW();
orderUoW.Create();
}
}
Notice that it creates and instance of my UoW?!? logic that it would hit on each pass.
OrderJobSchedule.cs logic:
public sealed class OrderJobSchedule
{
public void Start()
{
IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
scheduler.Start();
IJobDetail job = JobBuilder.Create<OrderJob>().Build();
ITrigger trigger = TriggerBuilder.Create()
.WithSimpleSchedule(a => a.WithIntervalInSeconds(15).RepeatForever())
.Build();
scheduler.ScheduleJob(job, trigger);
}
public void Stop()
{
IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
scheduler.Shutdown();
}
}
A lot of magic here, but to emphasis on:
JobBuilder.Create<OrderJob>().Build();
a.WithIntervalInSeconds(15).RepeatForever()
Now that's in place we need to add logic to the "guts" of the service:
public partial class QuayService : ServiceBase
{
OrderJobSchedule scheduler;
public QuayService()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
try
{
scheduler = new OrderJobSchedule();
scheduler.Start();
SendEmail("(Shopify)Quay Service Started",
"(Shopify)Quay Service Successfuly Started");
}
catch (Exception ex)
{
ProcessException(ex, "Error starting (Shopify)Quay Service");
EventLog.WriteEntry("Error starting (Shopify)Quay service and timer..." + ex.Message);
}
}
protected override void OnStop()
{
try
{
if (scheduler != null)
{
scheduler.Stop();
}
SendEmail("(Shopify)Quay Service stopped",
"(Shopify)Quay Service Successfuly Stopped");
}
catch (Exception ex)
{
ProcessException(ex, "Error stopping (Shopify)Quay Service");
EventLog.WriteEntry("Error stopping (Shopify)Quay timer and service..." + ex.Message);
}
}
private void SendEmail(string subject, string body)
{
new Email().SendErrorEmail("Quay", subject, body);
}
private void ProcessException(Exception ex,
string customMessage)
{
var innerException = "";
if (ex.InnerException != null)
innerException = (!string.IsNullOrWhiteSpace(ex.InnerException.Message)) ? ex.InnerException.Message : "";
new Email().SendErrorEmail("Quay", customMessage,
ex.Message + " " + innerException);
}
}
Very easy to set up and solved my horrendous experience with Timers.Timer Whilst I didn't fix the core issue, I came up with a solution and got a bug free working system in place.
Please note to future readers DO NOT USE System.Timers.Timer unless you are prepared to add a "hack'ish" fix.
I am coding a Forex Trading robot, and I am running an outOfMemory exception after some time (around 2 hours) using BlockingCollection.
I basically have 1 queue pair Trade chart, that are added into a dict:
private Dictionary<string, BlockingCollection<tick>> tickQueues = new Dictionary<string, BlockingCollection<tick>>();
I check the memory dump after one hour, and I can see the following items are piling up:
Count Size(bytes) Inclusive Size
ThreadPoolWorkQueue+QueueSegment 22,951 24,236,256 40,316,868
QueueUserWorkItemCallback 689,838 13,796,760 16,081,272
TimerQueueTimer 11,160 713,772 2,355,736
I have a timer that is responsible to add data to the Queue:
private void TickTimer_tick(object source, ElapsedEventArgs e) {
if (Monitor.TryEnter(LockTimerTick, GlobalSettings.APISleepDelayMSTick)) {
updateLockFailCount = 0;
try {
tick t = new tick(DateTime.Now, d.bid, d.ask);
lastBid = d.bid;
lastAsk = d.ask;
t.pair = Inst.pair;
//myTickQueue.TryAdd(t);
if (!myTickQueue.TryAdd(t)) {
functions.Logger.log("Error when adding Tick on Queue for " + Inst.pair+ " Maybe Queue is full", "SHMAPI", LOGLEVEL.WARN);
}
} catch (Exception E) {
functions.Logger.log("Error happened when refreshing tick data: " + E.Message, "SHMAPI", LOGLEVEL.ERROR);
} finally {
Monitor.Exit(LockTimerTick);
}
} else {
updateLockFailCount++;
int sev = LOGLEVEL.TRACE;
if (updateLockFailCount == 10) { sev = LOGLEVEL.DEBUG; }
if (updateLockFailCount==50) { sev = LOGLEVEL.WARN; }
if (updateLockFailCount % 100 == 0 && updateLockFailCount>=100) { sev = LOGLEVEL.ERROR; }
functions.Logger.log("Could not get lock to refresh tick data for symbol "+Symbol, "SHMAPI", sev);
}
}
And finally, my task that checks the Q:
public void startQueueTask(string Pair) {
if (!tickQueues.ContainsKey(Pair.ToUpper())) {
tickQueues.Add(Pair.ToUpper(), new BlockingCollection<tick>(GlobalSettings.tickQueueSize));
if (!MTAPIs.ContainsKey(Pair.ToUpper())) {
throw new Exception("API for pair " + Pair + " Should be initialized !!");
}
MTAPIs[Pair.ToUpper()].setTickQueue(tickQueues[Pair.ToUpper()]);
functions.Logger.log("Starting " + Pair + " Queue Task", "TICKPROCESSING", LOGLEVEL.DEBUG);
Task.Run(() => {
foreach (tick tick in tickQueues[Pair.ToUpper()].GetConsumingEnumerable()) {
try {
onTick(tick);
} catch (Exception E) {
functions.Logger.log("Error processing tick for symbol " + tick.pair + " " + E.Message, "TICKPROCESSING", LOGLEVEL.ERROR);
functions.printException(E);
}
}
functions.Logger.log("Exiting Queue Task", "TICKPROCESSING", LOGLEVEL.ERROR);
});
} else {
functions.Logger.log("Skipping " + Pair + " Queue Task because already exists", "TICKPROCESSING", LOGLEVEL.DEBUG);
}
}
I am not really sure why I am getting OOM, but it looks similar to:
http://blogs.microsoft.co.il/bnaya/2012/02/26/real-life-story-blocking-collection/
But I am not using parallel here... My Queues are empty though since week end market is closed.
Would using another timer with TryDequeue a better approach ?
Any advice would be welcome !
I switched timer to manual like this:
private void TickTimer_tick(object source, ElapsedEventArgs e) {
try {
//...
} finally {
TickTimer.Start();
}
}
And it seemed to have resolved my issue.
I am also making sure to send ticks in the Q, and they are filtered by duplicates by the Receiver, just so the Queing thread is never pending for too long as well.
Thanks for the pointer !