Consider the following code which uses basic Task library functionality with a CancellationTokenSource. It starts up a thread which fills a Dictionary with prices and reads the data from an SQL server db. The thread ends after about 10 minutes and every 2 hours it is fired up again, calling Cancel first in case the thread was still running.
private CancellationTokenSource mTokenSource = new CancellationTokenSource();
internal Prices(Dictionary<string, Dealer> dealers)
{
mDealers = dealers;
mTask = Task.Factory.StartNew
(() => ReadPrices(mTokenSource.Token), mTokenSource.Token);
}
internal void Cancel()
{
mTokenSource.Cancel();
}
private void ReadPrices(CancellationToken ct)
{
using (SqlConnection connection =
new SqlConnection(ConfigurationManager.AppSettings["DB"]))
{
connection.Open();
var dealerIds = from dealer in mDealers.Values
where dealer.Id != null
select dealer.Id;
foreach (var dealerId in dealerIds)
{
if (!ct.IsCancellationRequested)
{
FillPrices(connection);
}
else
break;
}
}
}
Now at some point the application crashes with the following exception in the event log.
Application: Engine.exe Framework Version: v4.0.30319 Description: The
process was terminated due to an unhandled exception. Exception Info:
System.AggregateException Stack: at
System.Threading.Tasks.TaskExceptionHolder.Finalize()
It must have to do with the code here because the Tasks library isn't used anywhere else but I cant figure out what is wrong with the code. Does anyone have an idea what could be wrong here?
Tasks like to feel listened to. It sounds like something isn't happy. You do get a "last chance" to hear it out, though:
TaskScheduler.UnobservedTaskException += (sender, args) =>
{
foreach (var ex in args.Exception.InnerExceptions)
{
Log(ex);
}
args.SetObserved();
};
Note this isn't intended as the fix - it is intended to let you see what Task is exploding and with what error. The SetObserved() will prevent it killing your application. But the fix here is ideally:
don't let your tasks throw,
or make sure that you're there to check the status of the task later
It is quite possibly not happy with your cancellation detection. IIRC the preferred way to do that would be:
foreach(...) {
if(ct.IsCancellationRequested) {
// any cleanup etc
ct.ThrowIfCancellationRequested();
}
...
}
or more simply if no cleanup is needed, just:
foreach(...) {
ct.ThrowIfCancellationRequested();
...
}
Equally, it could just be a data access exception. There are any number of exceptions that can happen when talking to a database. Timeout, deadlock, cannot-connect, etc.
Related
I'm at a loss. I have a Task factory that is starting a function:
Task.Factory.StartNew(() => ServerGetSecureLog(ServerAndProjectorArchive);
private static void ServerGetSecureLog(string archivePath)
{
var localClientManager = InitializeConnection();
var destinationPath = $"{ServerSecureLogFile}";
var result = ServerGetBasicSecureLogAsync(localClientManager);
var sw = StartStopwatch();
//Wait upto 120s
if (!result.Wait(TimeSpan.FromSeconds(120))) // <<< Exception thrown here.
{
StopStopwatch(sw);
log.Warn($"{MethodBase.GetCurrentMethod().Name} took too long to execute (timeout exceeded).");
}
else
{
StopStopwatch(sw);
Thing is, I'm getting an System.AggregateException being thrown as shown above. I must be going insane., because to debug this, I put breakpoints on EVERY line of code before that, on that same line the exception is being thrown and even the line that starts the thread and the lambda that is called, yet NONE of them are getting hit. Going up one stack frame and I get to the call of the lambda that I put the breakpoint on.
Variable state:
localClientManager looks to be uninitialized, so seems that the IP got hijacked somehow.
I wouldn't expect the optimizer would do anything on a DEBUG build and there are no unsafe areas in the code. What could cause such bazar behaviour?
Edit
Could Remote Debugging be an issue? Unfortunately, I can't put a debugger on that system, and can't really run the app from a local machine.
Whenever a Task run to completion in failed state - that is, when method that run that task somehow failed - it throws AggregateException.
This is due to the fact that a Task can be continued:
var t = SomeAsyncMethod().ContinueWith(a=> { });
If SomeAsyncMethod throws an exception and ContinueWith also throws an exception, the aggregated exception would have all exceptions for that call.
However, when you're using async/await calls and does try/catch, c# async state unrolls aggregated exceptions and throws original exception on your code.
The problem in your code is related with the fact that GetServerLog method is synchronous and is calling an Task method.
You can:
var localClientManager = InitializeConnection();
var destinationPath = $"{ServerSecureLogFile}";
//Wait upto 120s
var cts = new CancellationTokenSource(TimeSpan.FromSeconds(120));
// assumes that this method will look at the token to stop processing and bubbling up a OperationCancelledException/TimeoutException if token expires.
// localClientManager can be null if InitializeConnection returns null or depends on other lazy/async initialization steps.
var result = ServerGetBasicSecureLogAsync(localClientManager, cts.Token);
var sw = Stopwatch.StartNew();
try
{
// This can cause deadlocks it Thread pool threads are waiting for a free thread and other locations are doing Waits: https://stackoverflow.com/questions/13140523/await-vs-task-wait-deadlock
//result.WaitOne(TimeSpan.FromSeconds(120));
result.GetAwaiter().GetResult();
}
catch(MySimpleException)
{
}
catch(OperationCancelledException)
{
}
finally
{
sw.Stop();
// could enclose cts in using statement
cts.Dispose();
}
Because GetAwaiter().GetResult() is the same method called by async state, it also unrolls the aggregated exception and by using try/catch, you can look at operation cancelled as well.
Also, you don't need to create a Task to run an async method:
async Task SomeAsyncMethod()
{
var localClientManager = InitializeConnection();
var destinationPath = $"{ServerSecureLogFile}";
//Wait upto 120s
var cts = new CancellationTokenSource(TimeSpan.FromSeconds(120));
// assumes that this method will look at the token to stop processing and bubbling up a OperationCancelledException/TimeoutException if token expires.
// localClientManager can be null if InitializeConnection returns null or depends on other lazy/async initialization steps.
var sw = Stopwatch.StartNew();
try
{
await ServerGetBasicSecureLogAsync(localClientManager, cts.Token);
}
catch(MySimpleException)
{
}
catch(OperationCancelledException)
{
}
finally
{
sw.Stop();
// could enclose cts in using statement
cts.Dispose();
}
}
// this call will never throw as long as method is catching and handling exceptions
SomeAsyncMethod().GetAwaiter().GetResult();
I have a windows service with a listener that runs a loop within a task that listens to ServiceBus. If the connection goes down or some other problem occurs I want to start the listener task again. Since I do not do any await on the task as it is should run forever I do need to use ContinueWith and check if the exception has occurred. If it did I want to start exactly the same process.
So the question is: Is it safe to do it this way with regards to execution context, memory, stack trace or some other things I have not thought about?
The code seems to run all fine and reconnects if the network was down and came back online, stack trace for the exceptions seems correct as well but I am afraid of some pitfalls I have not thought about.
private void StartReceiving(string connectionString)
{
_receiverHost
.StartReceiving(connectionString)
.ContinueWith(c =>
{
if (c.IsFaulted)
{
Thread.Sleep(60000);
StartReceiving(connectionString);
}
}
});
}
To answer your question:
"Is it safe to do it this way with regards to execution context, memory, stack trace or some other things I have not thought about?"
Yes; there is no issues in regards of any of the mentioned areas that i can think of.
You basically exit the previous task and enter a new task. Your memory is cleaned when you continue the following task and the stacktrace is starting at the task creating the execution.
However I'd rather comment on the issues it introduces in regards of error handling and breaking the execution, which by this implementation does not help you to find the error, connect to an alternative address or some other logic you may introduce. In my opinion you introduce an unexpected complexity in this implementation.
If I had to implement this class, I would rather raise a disconnected event which the class using this object has to solve. This will give you a much wider range of opportunities, e.g. notifying other dependent classes.
Furthermore, freezing a thread is usually not a good solution. I'd rather start a timer or something similar.
I tried your code in some extra code as follows, and I wasn't able to see any issue with execution context, memory, stack trace.
public class Receiver
{
public async Task StartReceiving(string connectionString)
{
var task = Task.Run(() =>
{
try
{
throw new Exception("connection lost");
}
catch (Exception)
{
/* log the exception, or something */
throw;
}
});
await task;
}
}
public class Server
{
ILog log = LogManager.GetLogger<Server>();
public bool IsStopped { get; private set; } = false;
private Receiver _receiverHost = new Receiver();
public void StartReceiving(string connectionString)
{
_receiverHost
.StartReceiving(connectionString)
.ContinueWith(async c =>
{
if (c.IsFaulted)
{
var n = Process.GetCurrentProcess().Threads.Count;
var timestamp = DateTime.UtcNow;
await Task.Delay(1000);
log.Debug($"Task Delay: {(DateTime.UtcNow - timestamp).TotalSeconds} seconds");
StartReceiving(connectionString);
}
});
}
}
[TestFixture]
public class TestServerRetry
{
[TestCase]
public async Task TestRetry()
{
var server = new Server();
server.StartReceiving("test connection");
while (!server.IsStopped)
await Task.Delay(100);
}
}
I need to a background long running thread from the asp.net application.
BTW,the method fetches something from external data source period and maybe exception happens, I provide some ways to fullfill this task, please advice which way is best, please advice if there is better way.
Way1=Looping when exception happens.
static void LongRunningMethod()
{
do
{
try
{
//fetch something from external period and maybe exception happens.
Thread.sleep(100000);
}
catch(Exception ex)
{
//log exception..
}
} while (true);
}
Way2=Running the following method period by something like timer, and open a new thread when exception happens.
static void LongRunningMethod()
{
try
{
//fetch something from external period and maybe exception happens.
Thread.sleep(100000);
}
catch(Exception ex)
{
//log exception..
Thread T2 = new Thread(LongRunningMethod);
T2.Start();
}
}
Way3=Call itself when exception happens.
static void LongRunningMethod()
{
try
{
//fetch something from external period and maybe exception happens.
Thread.sleep(100000);
}
catch(Exception ex)
{
//log exception..
LongRunningMethod();
}
}
I will use none of the three. In fact, I seldom build interval tasks base on ASP.NET because:
1. The start of the task is out of control
Based on ASP.NET the task has to be started/registered on Application_Start method, which is triggered by the first request, that means your tasks is started when the first request comes. Theoretically it can be long time after your application is deployed.
2. The end of the task is out of control
The web servers (thinking IIS) can be configured as recycle-on-demand (AFAIK this is the default behavior). That is, your thread may be killed when executing. Under most circumstances you need to deal with the persistence of your tasks and states, and add retrying codes and etc... That's a nightmare!
For me it's a better idea to use a windows service, or the task scheduler function of windows, the latter is even easier because you don't need to write a timer or interval call codes. It's more stable, less code, and friendly logs in the event viewer! That's really make everything easier.
I like to use System.ComponentModel.BackgroundWorker:
static BackgroundWorker looper = new BackgroundWorker();
static bool isRunning = true;//you can set this to false when closing
public static void initialize(){
looper.DoWork+= doLoop;
looper.RunRunWorkerAsync();
}
private static void doLoop(object sender, DoWorkEventArgs e){
while(isRunning){
//do looping code
System.Threading.Thread.Sleep(5000);
if(!isRunning)
break;
}
}
PS - sorry for weird spacing - I edited this directly in the code window.
I have a long running Console application (3 hrs) which I need to kill with an Exception in certain circumstances. The reason for killing this batch is that it is monitored by operators and controlled by an enterprise scheduler. An unhandled exception will alert them and its message used by the on-call support.
Ideally, I want a background thread running as a TPL Task to check for these database defined criteria, and if encountered throw an exception to fail the process. Throwing an exception in a Task does not kill the main thread. I have tried the following, running a ContinueWith Task on the main's scheduler but it does not affect main.
static void Main(string[] args)
{
var mainThread = TaskScheduler.Current;
var taskDb = new Task<bool>(RunDb, TaskCreationOptions.LongRunning);
var taskHandle = taskDb.ContinueWith(b => Check(b.Result), mainThread);
taskDb.Start();
//do stuff
}
static bool RunDb()
{
//check database in loop
//if validation failed return false
return false;
}
static void Check(bool successful)
{
//check database in loop
//if validation failed return false
if(!successful)
throw new Exception("batch failed for these reasons...");
}
The only possible solution I've arrived at is modify a global property from within the ContinueWith action, and query this property from within the main thread, throwing an exception.
I've had a good look around the web and while there's plenty on cancelling threads from main, there's nothing about killing main from a background thread.
You can use Environment.FailFast everywhere to shutdown your application with an exception. This works from a background thread as well:
Task.Run(() => Environment.FailFast("Fail", new Exception("batch failed for these reasons...")));
However, I think a more robust design would be to have a system wide CancellationToken that you can cancel from your task that brings down the entire application gracefully.
Environment.Exit() is probably the easiest way. For example, the following program will exit after printing about 10 numbers, rather than the full 100.
public static void Main(string[] args)
{
Task.Run(() =>
{
Thread.Sleep(1000);
Console.Out.WriteLine("Something has gone terribly wrong!");
System.Environment.Exit(1);
});
for (int i = 0; i < 100; i++)
{
Console.Out.WriteLine(i);
Thread.Sleep(100);
}
}
I have some kind of data processing, that is depnded on the success of problematic method that returns a result that used in my processing.
The external method is problematic since it works slowly, it may crash and throw exceptions of any type, and I don't have its source code.
I want to use thread in the beginning of my processing to save some time, since my processing is long enough even without that problematic method. but there exists a point that I must have a valid result from the problematic method that I cannot continue if it fails.
I want the use the exceptions of the problematic method in the main thread, so they get the same exception handling as any other exceptions that may be thrown by my processing.
Here is my code - it seems ok and it works, but it just looks too cumbersome to me, so this my question: Is there a better approach to manage correctly the call to the problematic method by thread, and its potential exceptions?
My environment is .NET 3.5, so please I would like to get answers focusing that version, but I would like also to learn if there are new approaches for newer .NET versions.
Thank you very much!
public void DoProcess()
{
object locker = new object();
bool problematicCodeFinished = false;
Exception methodException = null;
Result result;
Func<Result> getProblematicResult = new Func<Result>(() => problematicMethod()); //create delegate
//run problematic method it in thread
getProblematicResult.BeginInvoke((ar) => //callback
{
lock(locker)
{
try
{
result = getProblematicResult.EndInvoke();
}
catch (Exception ex)
{
methodException = ex;
}
finally
{
problematicCodeFinished = true;
Monitor.Pulse(locker);
}
}
}, null);
//do more processing in main thread
// ...
// ...
// ...
try
{
//here we must know what was the result of the problematic method
lock (locker)
{
if (!problematicCodeFinished) //wait for problematic method to finish
{
Monitor.Wait(locker);
}
}
//throw problematic method exception in main thread
if (methodException != null)
{
throw methodException;
}
//if we are here we can assume that we have a valid result, continue processing
// ...
// ...
// ...
}
catch (Exception ex) //for the problematic method exception, or any other exception
{
//normal exception handling for my processing
}
}
You're making life hard for yourself. Try using tasks from the TPL instead.
Your code would look like this:
public void DoProcess()
{
var task = Task.Factory.StartNew(() => problematicMethod());
//do more processing in main thread
// ...
// ...
// ...
var result = task.Result;
}