My actor interacts with a non-Akka thing that has an async disposal. This disposal can take 5-10 seconds. I do this in PostStop() like so:
protected override void PostStop()
{
async Task DisposeThing()
{
Debug.WriteLine("Before Delay");
await Task.Delay(10000); // This would be the actual call to dispose the thing
Debug.WriteLine("After Delay");
};
ActorTaskScheduler.RunTask(async () =>
{
try
{
Debug.WriteLine("Before DisposeThing");
await DisposeThing();
Debug.WriteLine("After DisposeThing");
}
catch (Exception ex)
{
Debug.WriteLine($"An exception occured: {ex}");
}
finally
{
Debug.WriteLine("actor done disposing.");
}
});
base.PostStop();
}
Full gist here.
The parent does _childActor.Tell(PoisonPill.Instance). I also tried _childActor.GracefulStop with a large enough timeout.
In both cases, this prints:
Before DisposeThing
Before Delay
And that's it, the rest is never executed. Not even the finally executes (which I guess breaks C#? using doesn't work anymore, for instance).
Silently dropping await continuations (including finallys) could lead to some really tricky-to-understand bugs, so this leaves me with two questions:
when does Akka decide to simply drop an ongoing async function, is there a consistent model to be understood?
how should I write this in a way that is guaranteed to execute and not terminate the actor before disposal is done?
Update:
After sleeping on this I think I understand what's going on. Keep in mind that is mostly conjecture from someone that's been looking at Akka.Net for the past 2 days (e.g. this thread), and I post this because no one has answered yet.
The way Akka.Net implements async continuations is by having the actor executing the async function send ActorTaskSchedulerMessages to itself. This message points to the remaining work to be done after an await returns, and when the actor gets to process that message, it'll execute the continuation, up until the next await (or the end of the function if there no other await).
When you tell an actor to stop with a PoisonPill for instance, once that message is processed, no further messages are processed for that actor. This is fine when those messages are user-defined. However, this ends up also silently dropping any async continuations since they're also implemented as actor messages.
Indeed when running a program using the above actor, we can see this in the console:
[INFO][2022-01-11 2:59:43 PM][Thread 0004][akka://ActorSystem/user/$a/$a] Message [ActorTaskSchedulerMessage] from [akka://ActorSystem/user/$a/$a#132636847] to [akka://ActorSystem/user/$a/$a#132636847] was not delivered. 2 dead letters encountered. If this is not an expected behavior then [akka://ActorSystem/user/$a/$a#132636847] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
If this understanding is correct, this makes async extremely unreliable inside functions passed to ReceiveAsync, ActorTaskScheduler.RunTask etc. as you cannot ever assume anything after an await will get to execute, including exception handlers, cleanup code inside finallys, using statement disposal, etc. Actors can be killed stopped at any time.
I suppose then that since language primitives lose their meaning, what you need to do is wrap your Task-returning functions inside their own little actors and rely on Akka semantics rather than language semantics.
You captured what the issue was.
This is one solution I came up with:
using Akka.Actor;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using static Akka.NET_StackOverflow_Questions_tryout.Questions._70655287.ChildActor;
namespace Akka.NET_StackOverflow_Questions_tryout.Questions._70655287
{
public class ParentActor:ReceiveActor
{
private readonly IActorRef _child;
public ParentActor()
{
_child = Context.ActorOf(ChildActor.Prop());
Context.Watch(_child);
Receive<ShutDown>(s =>
{
_child.Forward(s);
});
Receive<Terminated>(t => {
var tt = t;
});
}
public static Props Prop()
{
return Props.Create(() => new ParentActor());
}
}
public class ChildActor : ReceiveActor
{
public ChildActor()
{
ReceiveAsync<ShutDown>(async _ =>
{
async Task DisposeThing()
{
Debug.WriteLine("Before Delay");
await Task.Delay(10000); // This would be the actual call to dispose the thing
Debug.WriteLine("After Delay");
};
await DisposeThing()
.ContinueWith(async task =>
{
if (task.IsFaulted || task.IsCanceled)
return; //you could notify the parent of this issue
await Self.GracefulStop(TimeSpan.FromSeconds(10));
});
});
}
protected override void PostStop()
{
base.PostStop();
}
public static Props Prop()
{
return Props.Create(()=> new ChildActor());
}
public sealed class ShutDown
{
public static ShutDown Instance => new ShutDown();
}
}
}
So instead of stopping the _childActor from the parentActor side you could send a shutdown message to the child to shutdown following the defined steps: first dispose the non-akka thing (to ensure it is truly not alive in-memory) afterwards, second, self-destruct the child which will notify the parent!
Related
I'm trying to design a public method that returns a quick result and, if needed, kicks off a long-running background task.
The result is retrieved within the method pretty quickly, and once it is retrieved it needs to be immediately available to the caller, without waiting for the potential background task to complete.
The background task is either not needed at all, or it must run for a significant amount of time - longer than the caller and all other logic in the program would take to complete without it.
Looking through the MS docs, the best design option I can come up with is to return two things from this method: the result, as well as a Task for the background task.
I don't see a better way to do this, but there are some downsides: first, the responsibility for making sure the background task completes falls on the caller of the method, even though the caller really just wants to consume the immediate result and not be concerned with the behind-the-scene stuff. Second, it is awkward to return null if the background task isn't needed to begin with: now the caller must ensure the task isn't null, in addition to making sure it completes if it is null.
What other options are available for this sort of situation?
Here's an example of what my current design looks like:
public async Task<Tuple<string, Task>> GetAndSave() {
var networkResult = await GetFromNetworkAsync();
if (NeedsToSave(networkResult)) {
var saveTask = SaveToDiskAsync(networkResult);
return new Tuple<string, Task>(networkResult, saveTask);
}
else {
return new Tuple<string, Task>(networkResult, null);
}
}
first, the responsibility for making sure the background task completes falls on the caller of the method, even though the caller really just wants to consume the immediate result and not be concerned with the behind-the-scene stuff.
If it's important to make sure the background task completes then instead of returning the Task you could hand it off to another object (that has been injected into the class that has your GetAndSave method). For example:
public class Foo
{
readonly Action<Task> _ensureCompletion;
public Foo(Action<Task> ensureCompletion)
{
_ensureCompletion = ensureCompletion;
}
public async Task<string> GetAndSaveAsync() // Your method
{
string networkResult = await GetFromNetworkAsync();
if (NeedsToSave(networkResult))
{
Task saveTask = SaveToDiskAsync(networkResult);
_ensureCompletion(saveTask); // This is a synchronous call... no await keyword here. And it returns `void`
}
return networkResult;
}
Task<string> GetFromNetworkAsync() {...}
bool NeedsToSave(string x) {...}
Task SaveToDiskAsync(string x) {...}
}
Now you can inject whatever follow-up behavior you desire for the saveTask. For example, you could write stuff out to the console depending on how it goes:
async Task DoStuff()
{
var foo = new Foo(async task =>
// ^^^^^ More on this in a moment
{
try
{
await task;
Console.Writeline("It worked!");
}
catch (Exception e)
{
Console.Writeline(e.ToString());
}
});
var immediateResult = await foo.GetAndSaveAsync();
// do stuff with `immediateResult`
}
Now, the thing that might be confusing about this (and the power behind this type of solution) is how you can have a synchronous call on the one hand:
_ensureCompletion(saveTask); // This is a synchronous call... no await keyword here
...that does asynchronous things:
var foo = new Foo(async task => ...);
// ^^^^^ This statement lambda is asynchronous
(The injected delegate might even write out to the console on a different thread than whatever thread called GetAndSaveAsync()!)
There's no magic here. It all comes down to SynchronizationContext and the inner workings of async/await.
When the compiler encounters the await keyword (in an async context) then it will do a few things:
Turn everything after the await into a continuation (basically a delegate with some state)
Replace the await keyword with something like SynchronizationContext.Current.Post(continuation)
In other words, this:
async void EnsureCompletion(Task task)
{
try
{
await task;
Console.Writeline("It worked!");
}
catch (Exception e)
{
Console.Writeline(e.ToString());
}
}
...gets turned into something like this:
void EnsureCompletion(Task task)
{
try
{
task.ContinueWith(t => SynchronizationContext.Current.Post(_ =>
{
if (t.IsCompletedSuccessfully)
{
Console.Writeline("It worked!");
}
else
{
Console.Writeline(task.Exception.ToString());
}
});
}
catch (Exception e)
{
Console.Writeline(e.ToString());
}
}
As you can see, EnsureCompletion is an entirely synchronous function that does (almost) the exact same thing as its asynchronous form. Notice how it returns void and everything. This is how you can jam an asynchronous statement lambda into a delegate parameter that has a synchronous signature.
I hinted that the console writing might happen on a totally different thread. That's because async/await is orthogonal to threading. It depends on whatever the currently assigned implementation of SynchronizationContext is programmed to do. Unless you're using WPF or WinForms then by default there will be no SynchronizationContext and continuations (the things that get passed to SynchronizationContext.Post) will just be tossed over to whatever thread happens to be free in the default thread pool.
Second, it is awkward to return null if the background task isn't needed to begin with: now the caller must ensure the task isn't null, in addition to making sure it completes if it is null.
I'm of the opinion that null was a design mistake in C#. (Ask me what some other ones are :) ). If you return null from this method:
public Task DoSomethingAsync()
{
return null; // Compiles just fine
}
...then anyone who uses it will encounter a NullReferenceException when they await it.
async Task Blah()
{
await DoSomethingAsync(); // Wham! NullReferenceException :(
}
(the reason why comes back to how the compiler desugurs the await keyword)
...and it's awkward to have to check for nulls everywhere.
Much better would be to just return Task.CompletedTask as #juharr said.
But if you just hand off the background task like I showed above then you don't have to worry about this.
I have a windows service with a listener that runs a loop within a task that listens to ServiceBus. If the connection goes down or some other problem occurs I want to start the listener task again. Since I do not do any await on the task as it is should run forever I do need to use ContinueWith and check if the exception has occurred. If it did I want to start exactly the same process.
So the question is: Is it safe to do it this way with regards to execution context, memory, stack trace or some other things I have not thought about?
The code seems to run all fine and reconnects if the network was down and came back online, stack trace for the exceptions seems correct as well but I am afraid of some pitfalls I have not thought about.
private void StartReceiving(string connectionString)
{
_receiverHost
.StartReceiving(connectionString)
.ContinueWith(c =>
{
if (c.IsFaulted)
{
Thread.Sleep(60000);
StartReceiving(connectionString);
}
}
});
}
To answer your question:
"Is it safe to do it this way with regards to execution context, memory, stack trace or some other things I have not thought about?"
Yes; there is no issues in regards of any of the mentioned areas that i can think of.
You basically exit the previous task and enter a new task. Your memory is cleaned when you continue the following task and the stacktrace is starting at the task creating the execution.
However I'd rather comment on the issues it introduces in regards of error handling and breaking the execution, which by this implementation does not help you to find the error, connect to an alternative address or some other logic you may introduce. In my opinion you introduce an unexpected complexity in this implementation.
If I had to implement this class, I would rather raise a disconnected event which the class using this object has to solve. This will give you a much wider range of opportunities, e.g. notifying other dependent classes.
Furthermore, freezing a thread is usually not a good solution. I'd rather start a timer or something similar.
I tried your code in some extra code as follows, and I wasn't able to see any issue with execution context, memory, stack trace.
public class Receiver
{
public async Task StartReceiving(string connectionString)
{
var task = Task.Run(() =>
{
try
{
throw new Exception("connection lost");
}
catch (Exception)
{
/* log the exception, or something */
throw;
}
});
await task;
}
}
public class Server
{
ILog log = LogManager.GetLogger<Server>();
public bool IsStopped { get; private set; } = false;
private Receiver _receiverHost = new Receiver();
public void StartReceiving(string connectionString)
{
_receiverHost
.StartReceiving(connectionString)
.ContinueWith(async c =>
{
if (c.IsFaulted)
{
var n = Process.GetCurrentProcess().Threads.Count;
var timestamp = DateTime.UtcNow;
await Task.Delay(1000);
log.Debug($"Task Delay: {(DateTime.UtcNow - timestamp).TotalSeconds} seconds");
StartReceiving(connectionString);
}
});
}
}
[TestFixture]
public class TestServerRetry
{
[TestCase]
public async Task TestRetry()
{
var server = new Server();
server.StartReceiving("test connection");
while (!server.IsStopped)
await Task.Delay(100);
}
}
I'd like to start off by saying I am in no means an expert in threading, and hence the reason for this question.
I have a home grown abstract logger which schedules calls to logging providers on a Task.Run like so:
public void Info(string message, Exception exception)
{
Task.Run(() =>
{
foreach (var p in _providers)
{
try
{
p.Info(message, exception);
}
catch
{
}
}
}).ConfigureAwait(false);
}
I am now running into a curious scenario where one of my logging providers is using an external library which writes to cosmos db and it does this, internally, on an asynchronous basis.
public void Debug<T>(string message, T objectToLog)
{
var model = GetLogModel(message, toSerialize: objectToLog);
_logRepo.Create(model); // << this guys is async
//var s= Task.Run(async () => await _logRepo.Create(model)).Result;
}
As you can see, by the commented out code, I'm currently trying some funky options to try and get this to run synchronously. I know I can call .Result without the secondary task below but that seems to be taboo.
My question is
How can I make this library code run synchronously without inadvertently causing deadlocks or other phantom problems that tend to be difficult to track down?
I am trying to create a logging library and things are good until the application shutdown is called. When application shutdown is called, any unfinished thread is killed and the specific log is lost.
As of now application exits even before the first 10 threads are complete. I want help on how to make the application wait until all threads created by library are done.
NOTE:
Requirement I got are like this. Modifications should be only in the class 'Logging' since this will be a library and will be provided to end users. Handling of logging issues during app shutdown must be done within it. This is where I have trouble now.
Alternatively a solution like create an event in logging class to trigger all logging complete, and ask user to call app exit on that event is possible, but that I am trying to avoid since it adds that burden to end user and adds complexity for implementations. There is a possibility they may skip it, which I do not want. I am looking for a solution like user should do 'Logging.AddException(....)' and then forget about it.
Please help. Provide comments if you are not clear about the idea.
Here is the full code abstract which you can put into a console application.
Note: Look for comments in CASE 1 and CASE 2.
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace MultithreadKeepAlive
{
class Program
{
static void Main(string[] args)
{
LogLoadTest();
Logging.AddExceptionEntryAsync(new Exception("Last Exception"));
/*
* USE CASE 1: Enable the below lines and you will see how long it is supposed to take.
* Notice that currentDomain_ProcessExit will not trigger if below gets uncommented
*/
//Console.WriteLine("Main thread wait override");
//Console.ReadLine();
}
static void LogLoadTest()
{
//In real world this will be called from any place of application like startup or just after application shutdown is initiated.
//: NOTICE: Unlike the sample here, this will never be on loop and I am not looking for handling multithreads in this class.
// That responsibility I am planning to assign to Logging class.
// AND ALSO the class Logging is going to be in a seperate signed assembly where user of this class ('Program') should not worry about multithreads.
Task t;
for (int i = 0; i < 40; i++)
{
t = Logging.AddExceptionEntryAsync(new Exception("Hello Exception " + i), "Header info" + i);
}
}
}
public class Logging
{
static List<Task> tasks = new List<Task>();
static AppDomain currentDomain;
static Logging()
{
currentDomain = AppDomain.CurrentDomain;
currentDomain.ProcessExit += currentDomain_ProcessExit;
}
public static async Task AddExceptionEntryAsync(Exception ex, string header = "")
{
Task t = Task.Factory.StartNew(() => AddExceptionEntry(ex, header));
tasks.Add(t);
await t;
}
public static void AddExceptionEntry(Exception ex, string header)
{
/* Exception processing and write to file or DB. This might endup in file locks or
* network or any other cases where it will take delays from 1 sec to 5 minutes. */
Thread.Sleep(new Random().Next(1, 1000));
Console.WriteLine(ex.Message);
}
static void currentDomain_ProcessExit(object sender, EventArgs e)
{
Console.WriteLine("Application shutdown triggerd just now.");
Process.GetCurrentProcess().WaitForExit(); //1st attempt.
//Task.WaitAll(tasks.ToArray()); //2nd attempt
while (tasks.Any(t => !t.IsCompleted)) //3rd attempt.
{
}
/* USE CASE 2: IF WORKING GOOD, THIS WILL BE DISPLAYED IN CONSOLE AS LAST
* MESSAGE OF APPLICATION AND WILL WAIT FOR USER. THIS IS NOT WORKING NOW.*/
Console.WriteLine("All complete"); //this message should show up if this work properly
Console.ReadLine(); //for testing purpose wait for input from user after every thread is complete. Check all 40 threads are in console.
}
}
}
You can try
Task.WaitAll(tasks);
This waits for all of the provided Task objects to complete execution.
UPDATE : using async/await
With async and await, we formalize and clarify how asynchronous, non-blocking methods begin and end. An async method can return only void or a Task.
static void Main()
{
// Create task and start it.
// ... Wait for it to complete.
Task task = new Task(AsyncMethod);
task.Start();
task.Wait();
}
public static async void AsyncMethod(){
await AnotherMehod();}
static async Task AnotherMehod() { //TODO}
As of now I myself found a workaround.
/// <summary>
/// Makes the current thread Wait until any of the pending messages/Exceptions/Logs are completly written into respective sources.
/// Call this method before application is shutdown to make sure all logs are saved properly.
/// </summary>
public static void WaitForLogComplete()
{
Task.WaitAll(tasks.Values.ToArray());
}
Step 1: Consider changing to Task.Run() if you do not want the scheduler to be involved. I'm also assuming you want to wait until all async tasks finish.
public static AddExceptionEntry(Exception ex, string header = "")
{
Task t = Task.Factory.StartNew(() => AddExceptionEntry(ex, header));
tasks.Add(t);
WaitForExecutionAsync().ConfigureAwait(true);
}
public static async Task WaitForExecutionAsync()
{
if(tasks.Count >0)
await Task.WhenAll(tasks.ToArray());
// Raise Event.
}
To Block just call this to run sync vs async:
WaitForExecution().GetAwaiter().GetResult();
I've tried to read up on async methods and am now trying to create my own async method. The method is a webservice call that returns a list of error logs. I'm not sure that I've understood correctly so I thought I'd share my code to see if I should do anything different.
All I want the code to do is return a list of errorlogs by calling a method GetAllErrorLogs(), that is a synchronized method. Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method. Here is the code.
[WebMethod]
public async Task<List<ErrorLog>> GetAllErrorLogs()
{
List<ErrorLog> errorLogs = new List<ErrorLog>();
await System.Threading.Tasks.Task.Run(() => {
errorLogs = ErrorLogRepository.GetAllErrorLogs();
});
if (errorLogs == null)
return new List<ErrorLog>();
return errorLogs;
}
Thanks!
I recently gave a talk at ThatConference on async on the server side, and I address this issue in the slides.
On the server side, you want to avoid the use of Task.Run and other constructs that queue work to the thread pool. As much as possible, keep thread pool threads available for handling requests.
So, ideally your repository would have an asynchronous method GetAllErrorLogsAsync, which would itself be asynchronous. If GetAllErrorLogs cannot be asynchronous, then you may as well just call it directly (removing the await Task.Run).
Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method.
If you have a GetAllErrorLogsAsync available, then this can easily be done using Task.WhenAll. However, if GetAllErrorLogs is synchronous, then you can only do this by doing parallel work in your request (e.g., multiple calls to Task.Run followed by Task.WhenAll).
Parallel code on the server must be approached with great trepidation. It is only acceptable in a very limited set of scenarios. The entire point of async on the server side is to use fewer threads per request, and when you start parallelizing, you're doing the opposite: multiple threads per request. This is only appropriate if you know your user base is very small; otherwise, you'll kill your server scalability.
I found this great codeproject detailed article about how to achieve that
http://www.codeproject.com/Articles/600926/Asynchronous-web-services-call-in-ASP-NET
**This is potentially wrong, read comments or spinoff question at HttpContext.Current after an await
If ErrorLogRepository.GetAllErrorLogs() is not thread-safe, it will cause weird bugs and potentially exception out. Make sure your code is ready for multi-threaded operation before switching to async methods, this is obviously very trivial advice but often overlooked. For example, if you reference HttpContext.Current in your methods, your code will die in the async method, and sometimes even AFTER the await. The reason is that the code within the async block will potentially be run on a separate thread, which will not have access to the same HttpContext.Current thread-static property, and await gets compiled into two methods. All code before an await gets run on one thread, and then calls the code after an await keyword as a continuation, but potentially on yet another thread. So sometimes your code will even work in an async block, only to choke unexpectedly after it gets "out" of the async back to what you think is a synchronous part of your code (but in reality everything after an await keyword is already not guaranteed to be the original thread).
Here is some production code...
using System.Web.Http;
using AysncTask = System.Threading.Tasks.Task;
public class myController : ApiControllerBase
{
[HttpPut]
[Route("api/cleardata/{id}/{requestId}/")]
public async AysncTask ClearData(Guid id, Guid requestId)
{
try
{
await AysncTask.Run(() => DoClearData(id, requestId));
}
catch (Exception ex)
{
throw new Exception("Exception in myController.ClearData", ex);
}
}
}
Handling Async exceptions is also VERY VERY important.. although this is for a windows console app, the same principles should apply.
source: https://blogs.msdn.microsoft.com/ptorr/2014/12/10/async-exceptions-in-c/
using System;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
namespace AsyncAndExceptions
{
class Program
{
static void Main(string[] args)
{
AppDomain.CurrentDomain.UnhandledException += (s, e) => Log("*** Crash! ***", "UnhandledException");
TaskScheduler.UnobservedTaskException += (s, e) => Log("*** Crash! ***", "UnobservedTaskException");
RunTests();
// Let async tasks complete...
Thread.Sleep(500);
GC.Collect(3, GCCollectionMode.Forced, true);
}
private static async Task RunTests()
{
try
{
// crash
// _1_VoidNoWait();
// crash
// _2_AsyncVoidAwait();
// OK
// _3_AsyncVoidAwaitWithTry();
// crash - no await
// _4_TaskNoWait();
// crash - no await
// _5_TaskAwait();
// OK
// await _4_TaskNoWait();
// OK
// await _5_TaskAwait();
}
catch (Exception ex) { Log("Exception handled OK"); }
// crash - no try
// await _4_TaskNoWait();
// crash - no try
// await _5_TaskAwait();
}
// Unsafe
static void _1_VoidNoWait()
{
ThrowAsync();
}
// Unsafe
static async void _2_AsyncVoidAwait()
{
await ThrowAsync();
}
// Safe
static async void _3_AsyncVoidAwaitWithTry()
{
try { await ThrowAsync(); }
catch (Exception ex) { Log("Exception handled OK"); }
}
// Safe only if caller uses await (or Result) inside a try
static Task _4_TaskNoWait()
{
return ThrowAsync();
}
// Safe only if caller uses await (or Result) inside a try
static async Task _5_TaskAwait()
{
await ThrowAsync();
}
// Helper that sets an exception asnychronously
static Task ThrowAsync()
{
TaskCompletionSource tcs = new TaskCompletionSource();
ThreadPool.QueueUserWorkItem(_ => tcs.SetException(new Exception("ThrowAsync")));
return tcs.Task;
}
internal static void Log(string message, [CallerMemberName] string caller = "")
{
Console.WriteLine("{0}: {1}", caller, message);
}
}
}