I'd like to start off by saying I am in no means an expert in threading, and hence the reason for this question.
I have a home grown abstract logger which schedules calls to logging providers on a Task.Run like so:
public void Info(string message, Exception exception)
{
Task.Run(() =>
{
foreach (var p in _providers)
{
try
{
p.Info(message, exception);
}
catch
{
}
}
}).ConfigureAwait(false);
}
I am now running into a curious scenario where one of my logging providers is using an external library which writes to cosmos db and it does this, internally, on an asynchronous basis.
public void Debug<T>(string message, T objectToLog)
{
var model = GetLogModel(message, toSerialize: objectToLog);
_logRepo.Create(model); // << this guys is async
//var s= Task.Run(async () => await _logRepo.Create(model)).Result;
}
As you can see, by the commented out code, I'm currently trying some funky options to try and get this to run synchronously. I know I can call .Result without the secondary task below but that seems to be taboo.
My question is
How can I make this library code run synchronously without inadvertently causing deadlocks or other phantom problems that tend to be difficult to track down?
Related
I am currently working on building a scanning portion of an app, and noticed that when my eventlistener hits and executes a function, I eventually get the following error:
A second operation was started on this context instance before a previous operation completed. This is usually caused by different threads concurrently using the same instance of DbContext.
After doing some research here, I learned that I should probably be awaiting those processes, but what I've added doesn't seem to be doing the trick. My threads are still getting tangled. My console shows the issue occuring specifically at:
GetByBarcode(String barcode)
But I'm not sure where else I would need to account for an await operator in that task. Am I missing an await call somewhere, or could the issue actually big bigger than that? Perhaps I've misunderstood await. I've added the relevant portions of code below.
<div #onkeydown="#KeyDown"
tabindex="0"
#ref="scanner">
public async Task KeyDown(KeyboardEventArgs e)
{
barcode += e.Key;
await CheckBarcode();
}
private async Task CheckBarcode()
{
testStuff = await stuffRepository.GetByBarcode(barcode);
if(testStuff is null)
{
//
}
else
{
await stuffRepository.Update(testStuff, loginID);
await LoadStuff();
barcode = "";
}
}
public async Task<StuffDTO> GetByBarcode(string barcode)
{
var obj = await _db.StuffTable.FirstOrDefaultAsync(u => u.Barcode == barcode);
if (obj != null)
{
return _mapper.Map<Stuff, StuffDTO>(obj);
}
return new StuffDTO();
}
In the async world you can be awaiting one DbContext operation and trying to start another at the same time. That's what the error is telling you.
You should switch to using the DbContextFactory and separate contexts per operation. The DbContextFactory manages a pool of connections and deals with create and disposing of individual contexts. See this MS article that explains how to configure it - https://learn.microsoft.com/en-us/ef/core/dbcontext-configuration/#using-a-dbcontext-factory-eg-for-blazor.
You should also be building some form of data pipeline to interact with your database, not using Db contexts directly in your components.
My actor interacts with a non-Akka thing that has an async disposal. This disposal can take 5-10 seconds. I do this in PostStop() like so:
protected override void PostStop()
{
async Task DisposeThing()
{
Debug.WriteLine("Before Delay");
await Task.Delay(10000); // This would be the actual call to dispose the thing
Debug.WriteLine("After Delay");
};
ActorTaskScheduler.RunTask(async () =>
{
try
{
Debug.WriteLine("Before DisposeThing");
await DisposeThing();
Debug.WriteLine("After DisposeThing");
}
catch (Exception ex)
{
Debug.WriteLine($"An exception occured: {ex}");
}
finally
{
Debug.WriteLine("actor done disposing.");
}
});
base.PostStop();
}
Full gist here.
The parent does _childActor.Tell(PoisonPill.Instance). I also tried _childActor.GracefulStop with a large enough timeout.
In both cases, this prints:
Before DisposeThing
Before Delay
And that's it, the rest is never executed. Not even the finally executes (which I guess breaks C#? using doesn't work anymore, for instance).
Silently dropping await continuations (including finallys) could lead to some really tricky-to-understand bugs, so this leaves me with two questions:
when does Akka decide to simply drop an ongoing async function, is there a consistent model to be understood?
how should I write this in a way that is guaranteed to execute and not terminate the actor before disposal is done?
Update:
After sleeping on this I think I understand what's going on. Keep in mind that is mostly conjecture from someone that's been looking at Akka.Net for the past 2 days (e.g. this thread), and I post this because no one has answered yet.
The way Akka.Net implements async continuations is by having the actor executing the async function send ActorTaskSchedulerMessages to itself. This message points to the remaining work to be done after an await returns, and when the actor gets to process that message, it'll execute the continuation, up until the next await (or the end of the function if there no other await).
When you tell an actor to stop with a PoisonPill for instance, once that message is processed, no further messages are processed for that actor. This is fine when those messages are user-defined. However, this ends up also silently dropping any async continuations since they're also implemented as actor messages.
Indeed when running a program using the above actor, we can see this in the console:
[INFO][2022-01-11 2:59:43 PM][Thread 0004][akka://ActorSystem/user/$a/$a] Message [ActorTaskSchedulerMessage] from [akka://ActorSystem/user/$a/$a#132636847] to [akka://ActorSystem/user/$a/$a#132636847] was not delivered. 2 dead letters encountered. If this is not an expected behavior then [akka://ActorSystem/user/$a/$a#132636847] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
If this understanding is correct, this makes async extremely unreliable inside functions passed to ReceiveAsync, ActorTaskScheduler.RunTask etc. as you cannot ever assume anything after an await will get to execute, including exception handlers, cleanup code inside finallys, using statement disposal, etc. Actors can be killed stopped at any time.
I suppose then that since language primitives lose their meaning, what you need to do is wrap your Task-returning functions inside their own little actors and rely on Akka semantics rather than language semantics.
You captured what the issue was.
This is one solution I came up with:
using Akka.Actor;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using static Akka.NET_StackOverflow_Questions_tryout.Questions._70655287.ChildActor;
namespace Akka.NET_StackOverflow_Questions_tryout.Questions._70655287
{
public class ParentActor:ReceiveActor
{
private readonly IActorRef _child;
public ParentActor()
{
_child = Context.ActorOf(ChildActor.Prop());
Context.Watch(_child);
Receive<ShutDown>(s =>
{
_child.Forward(s);
});
Receive<Terminated>(t => {
var tt = t;
});
}
public static Props Prop()
{
return Props.Create(() => new ParentActor());
}
}
public class ChildActor : ReceiveActor
{
public ChildActor()
{
ReceiveAsync<ShutDown>(async _ =>
{
async Task DisposeThing()
{
Debug.WriteLine("Before Delay");
await Task.Delay(10000); // This would be the actual call to dispose the thing
Debug.WriteLine("After Delay");
};
await DisposeThing()
.ContinueWith(async task =>
{
if (task.IsFaulted || task.IsCanceled)
return; //you could notify the parent of this issue
await Self.GracefulStop(TimeSpan.FromSeconds(10));
});
});
}
protected override void PostStop()
{
base.PostStop();
}
public static Props Prop()
{
return Props.Create(()=> new ChildActor());
}
public sealed class ShutDown
{
public static ShutDown Instance => new ShutDown();
}
}
}
So instead of stopping the _childActor from the parentActor side you could send a shutdown message to the child to shutdown following the defined steps: first dispose the non-akka thing (to ensure it is truly not alive in-memory) afterwards, second, self-destruct the child which will notify the parent!
I have a windows service with a listener that runs a loop within a task that listens to ServiceBus. If the connection goes down or some other problem occurs I want to start the listener task again. Since I do not do any await on the task as it is should run forever I do need to use ContinueWith and check if the exception has occurred. If it did I want to start exactly the same process.
So the question is: Is it safe to do it this way with regards to execution context, memory, stack trace or some other things I have not thought about?
The code seems to run all fine and reconnects if the network was down and came back online, stack trace for the exceptions seems correct as well but I am afraid of some pitfalls I have not thought about.
private void StartReceiving(string connectionString)
{
_receiverHost
.StartReceiving(connectionString)
.ContinueWith(c =>
{
if (c.IsFaulted)
{
Thread.Sleep(60000);
StartReceiving(connectionString);
}
}
});
}
To answer your question:
"Is it safe to do it this way with regards to execution context, memory, stack trace or some other things I have not thought about?"
Yes; there is no issues in regards of any of the mentioned areas that i can think of.
You basically exit the previous task and enter a new task. Your memory is cleaned when you continue the following task and the stacktrace is starting at the task creating the execution.
However I'd rather comment on the issues it introduces in regards of error handling and breaking the execution, which by this implementation does not help you to find the error, connect to an alternative address or some other logic you may introduce. In my opinion you introduce an unexpected complexity in this implementation.
If I had to implement this class, I would rather raise a disconnected event which the class using this object has to solve. This will give you a much wider range of opportunities, e.g. notifying other dependent classes.
Furthermore, freezing a thread is usually not a good solution. I'd rather start a timer or something similar.
I tried your code in some extra code as follows, and I wasn't able to see any issue with execution context, memory, stack trace.
public class Receiver
{
public async Task StartReceiving(string connectionString)
{
var task = Task.Run(() =>
{
try
{
throw new Exception("connection lost");
}
catch (Exception)
{
/* log the exception, or something */
throw;
}
});
await task;
}
}
public class Server
{
ILog log = LogManager.GetLogger<Server>();
public bool IsStopped { get; private set; } = false;
private Receiver _receiverHost = new Receiver();
public void StartReceiving(string connectionString)
{
_receiverHost
.StartReceiving(connectionString)
.ContinueWith(async c =>
{
if (c.IsFaulted)
{
var n = Process.GetCurrentProcess().Threads.Count;
var timestamp = DateTime.UtcNow;
await Task.Delay(1000);
log.Debug($"Task Delay: {(DateTime.UtcNow - timestamp).TotalSeconds} seconds");
StartReceiving(connectionString);
}
});
}
}
[TestFixture]
public class TestServerRetry
{
[TestCase]
public async Task TestRetry()
{
var server = new Server();
server.StartReceiving("test connection");
while (!server.IsStopped)
await Task.Delay(100);
}
}
I've tried to read up on async methods and am now trying to create my own async method. The method is a webservice call that returns a list of error logs. I'm not sure that I've understood correctly so I thought I'd share my code to see if I should do anything different.
All I want the code to do is return a list of errorlogs by calling a method GetAllErrorLogs(), that is a synchronized method. Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method. Here is the code.
[WebMethod]
public async Task<List<ErrorLog>> GetAllErrorLogs()
{
List<ErrorLog> errorLogs = new List<ErrorLog>();
await System.Threading.Tasks.Task.Run(() => {
errorLogs = ErrorLogRepository.GetAllErrorLogs();
});
if (errorLogs == null)
return new List<ErrorLog>();
return errorLogs;
}
Thanks!
I recently gave a talk at ThatConference on async on the server side, and I address this issue in the slides.
On the server side, you want to avoid the use of Task.Run and other constructs that queue work to the thread pool. As much as possible, keep thread pool threads available for handling requests.
So, ideally your repository would have an asynchronous method GetAllErrorLogsAsync, which would itself be asynchronous. If GetAllErrorLogs cannot be asynchronous, then you may as well just call it directly (removing the await Task.Run).
Since it can take a second to fetch all the error logs I want to have the opportunity to do other stuff once I called the GetAllErrorLogs() method.
If you have a GetAllErrorLogsAsync available, then this can easily be done using Task.WhenAll. However, if GetAllErrorLogs is synchronous, then you can only do this by doing parallel work in your request (e.g., multiple calls to Task.Run followed by Task.WhenAll).
Parallel code on the server must be approached with great trepidation. It is only acceptable in a very limited set of scenarios. The entire point of async on the server side is to use fewer threads per request, and when you start parallelizing, you're doing the opposite: multiple threads per request. This is only appropriate if you know your user base is very small; otherwise, you'll kill your server scalability.
I found this great codeproject detailed article about how to achieve that
http://www.codeproject.com/Articles/600926/Asynchronous-web-services-call-in-ASP-NET
**This is potentially wrong, read comments or spinoff question at HttpContext.Current after an await
If ErrorLogRepository.GetAllErrorLogs() is not thread-safe, it will cause weird bugs and potentially exception out. Make sure your code is ready for multi-threaded operation before switching to async methods, this is obviously very trivial advice but often overlooked. For example, if you reference HttpContext.Current in your methods, your code will die in the async method, and sometimes even AFTER the await. The reason is that the code within the async block will potentially be run on a separate thread, which will not have access to the same HttpContext.Current thread-static property, and await gets compiled into two methods. All code before an await gets run on one thread, and then calls the code after an await keyword as a continuation, but potentially on yet another thread. So sometimes your code will even work in an async block, only to choke unexpectedly after it gets "out" of the async back to what you think is a synchronous part of your code (but in reality everything after an await keyword is already not guaranteed to be the original thread).
Here is some production code...
using System.Web.Http;
using AysncTask = System.Threading.Tasks.Task;
public class myController : ApiControllerBase
{
[HttpPut]
[Route("api/cleardata/{id}/{requestId}/")]
public async AysncTask ClearData(Guid id, Guid requestId)
{
try
{
await AysncTask.Run(() => DoClearData(id, requestId));
}
catch (Exception ex)
{
throw new Exception("Exception in myController.ClearData", ex);
}
}
}
Handling Async exceptions is also VERY VERY important.. although this is for a windows console app, the same principles should apply.
source: https://blogs.msdn.microsoft.com/ptorr/2014/12/10/async-exceptions-in-c/
using System;
using System.Runtime.CompilerServices;
using System.Threading;
using System.Threading.Tasks;
namespace AsyncAndExceptions
{
class Program
{
static void Main(string[] args)
{
AppDomain.CurrentDomain.UnhandledException += (s, e) => Log("*** Crash! ***", "UnhandledException");
TaskScheduler.UnobservedTaskException += (s, e) => Log("*** Crash! ***", "UnobservedTaskException");
RunTests();
// Let async tasks complete...
Thread.Sleep(500);
GC.Collect(3, GCCollectionMode.Forced, true);
}
private static async Task RunTests()
{
try
{
// crash
// _1_VoidNoWait();
// crash
// _2_AsyncVoidAwait();
// OK
// _3_AsyncVoidAwaitWithTry();
// crash - no await
// _4_TaskNoWait();
// crash - no await
// _5_TaskAwait();
// OK
// await _4_TaskNoWait();
// OK
// await _5_TaskAwait();
}
catch (Exception ex) { Log("Exception handled OK"); }
// crash - no try
// await _4_TaskNoWait();
// crash - no try
// await _5_TaskAwait();
}
// Unsafe
static void _1_VoidNoWait()
{
ThrowAsync();
}
// Unsafe
static async void _2_AsyncVoidAwait()
{
await ThrowAsync();
}
// Safe
static async void _3_AsyncVoidAwaitWithTry()
{
try { await ThrowAsync(); }
catch (Exception ex) { Log("Exception handled OK"); }
}
// Safe only if caller uses await (or Result) inside a try
static Task _4_TaskNoWait()
{
return ThrowAsync();
}
// Safe only if caller uses await (or Result) inside a try
static async Task _5_TaskAwait()
{
await ThrowAsync();
}
// Helper that sets an exception asnychronously
static Task ThrowAsync()
{
TaskCompletionSource tcs = new TaskCompletionSource();
ThreadPool.QueueUserWorkItem(_ => tcs.SetException(new Exception("ThrowAsync")));
return tcs.Task;
}
internal static void Log(string message, [CallerMemberName] string caller = "")
{
Console.WriteLine("{0}: {1}", caller, message);
}
}
}
With business logic encapsulated behind synchronous service calls e.g.:
interface IFooService
{
Foo GetFooById(int id);
int SaveFoo(Foo foo);
}
What is the best way to extend/use these service calls in an asynchronous fashion?
At present I've created a simple AsyncUtils class:
public static class AsyncUtils
{
public static void Execute<T>(Func<T> asyncFunc)
{
Execute(asyncFunc, null, null);
}
public static void Execute<T>(Func<T> asyncFunc, Action<T> successCallback)
{
Execute(asyncFunc, successCallback, null);
}
public static void Execute<T>(Func<T> asyncFunc, Action<T> successCallback, Action<Exception> failureCallback)
{
ThreadPool.UnsafeQueueUserWorkItem(state => ExecuteAndHandleError(asyncFunc, successCallback, failureCallback), null);
}
private static void ExecuteAndHandleError<T>(Func<T> asyncFunc, Action<T> successCallback, Action<Exception> failureCallback)
{
try
{
T result = asyncFunc();
if (successCallback != null)
{
successCallback(result);
}
}
catch (Exception e)
{
if (failureCallback != null)
{
failureCallback(e);
}
}
}
}
Which lets me call anything asynchronously:
AsyncUtils(
() => _fooService.SaveFoo(foo),
id => HandleFooSavedSuccessfully(id),
ex => HandleFooSaveError(ex));
Whilst this works in simple use cases it quickly gets tricky if other processes need to coordinate about the results, for example if I need to save three objects asynchronously before the current thread can continue then I'd like a way to wait-on/join the worker threads.
Options I've thought of so far include:
having AsyncUtils return a WaitHandle
having AsyncUtils use an AsyncMethodCaller and return an IAsyncResult
rewriting the API to include Begin, End async calls
e.g. something resembling:
interface IFooService
{
Foo GetFooById(int id);
IAsyncResult BeginGetFooById(int id);
Foo EndGetFooById(IAsyncResult result);
int SaveFoo(Foo foo);
IAsyncResult BeginSaveFoo(Foo foo);
int EndSaveFoo(IAsyncResult result);
}
Are there other approaches I should consider? What are the benefits and potential pitfalls of each?
Ideally I'd like to keep the service layer simple/synchronous and provide some easy to use utility methods for calling them asynchronously. I'd be interested in hearing about solutions and ideas applicable to C# 3.5 and C# 4 (we haven't upgraded yet but will do in the relatively near future).
Looking forward to your ideas.
Given your requirement to stay .NET 2.0 only, and not work on 3.5 or 4.0, this is probably the best option.
I do have three remarks on your current implementation.
Is there a specific reason you're using ThreadPool.UnsafeQueueUserWorkItem? Unless there is a specific reason this is required, I would recommend using ThreadPool.QueueUserWorkItem instead, especially if you're in a large development team. The Unsafe version can potentially allow security flaws to appear as you lose the calling stack, and as a result, the ability to control permissions as closely.
The current design of your exception handling, using the failureCallback, will swallow all exceptions, and provide no feedback, unless a callback is defined. It might be better to propogate the exception and let it bubble up if you're not going to handle it properly. Alternatively, you could push this back onto the calling thread in some fashion, though this would require using something more like IAsyncResult.
You currently have no way to tell if an asynchronous call is completed. This would be the other advantage of using IAsyncResult in your design (though it does add some complexity to the implementation).
Once you upgrade to .NET 4, however, I would recommend just putting this in a Task or Task<T>, as it was designed to handle this very cleanly. Instead of:
AsyncUtils(
() => _fooService.SaveFoo(foo),
id => HandleFooSavedSuccessfully(id),
ex => HandleFooSaveError(ex));
You can use the built-in tools and just write:
var task = Task.Factory.StartNew(
() => return _fooService.SaveFoo(foo) );
task.ContinueWith(
t => HandleFooSavedSuccessfully(t.Result),
TaskContinuationOptions.NotOnFaulted);
task.ContinueWith(
t => try { t.Wait(); } catch( Exception e) { HandleFooSaveError(e); },
TaskContinuationOptions.OnlyOnFaulted );
Granted, the last line there is a bit odd, but that's mainly because I tried to keep your existing API. If you reworked it a bit, you could simplify it...
Asynchronous interface (based on IAsyncResult) is useful only when you have some non-blocking call under the cover. The main point of the interface is to make it possible to do the call without blocking the caller thread.
This is useful in scenarios when you can make some system call and the system will notify you back when something happens (e.g. when a HTTP response is received or when an event happens).
The price for using IAsyncResult based interface is that you have to write code in a somewhat awkward way (by making every call using callback). Even worse, asynchronous API makes it impossible to use standard language constructs like while, for, or try..catch.
I don't really see the point of wrapping synchronous API into asynchronous interface, because you won't get the benefit (there will always be some thread blocked) and you'll just get more awkward way of calling it.
Of course, it makes a perfect sense to run the synchronous code on a background thread somehow (to avoid blocking the main application thread). Either using Task<T> on .NET 4.0 or using QueueUserWorkItem on .NET 2.0. However, I'm not sure if this should be done automatically in the service - it feels like doing this on the caller side would be easier, because you may need to perform multiple calls to the service. Using asynchronous API, you'd have to write something like:
svc.BeginGetFooId(ar1 => {
var foo = ar1.Result;
foo.Prop = 123;
svc.BeginSaveFoo(foo, ar2 => {
// etc...
}
});
When using synchronous API, you'd write something like:
ThreadPool.QueueUserWorkItem(() => {
var foo = svc.GetFooId();
foo.Prop = 123;
svc.SaveFoo(foo);
});
The following is a response to Reed's follow-up question. I'm not suggesting that it's the right way to go.
public static int PerformSlowly(int id)
{
// Addition isn't so hard, but let's pretend.
Thread.Sleep(10000);
return 42 + id;
}
public static Task<int> PerformTask(int id)
{
// Here's the straightforward approach.
return Task.Factory.StartNew(() => PerformSlowly(id));
}
public static Lazy<int> PerformLazily(int id)
{
// Start performing it now, but don't block.
var task = PerformTask(id);
// JIT for the value being checked, block and retrieve.
return new Lazy<int>(() => task.Result);
}
static void Main(string[] args)
{
int i;
// Start calculating the result, using a Lazy<int> as the future value.
var result = PerformLazily(7);
// Do assorted work, then get result.
i = result.Value;
// The alternative is to use the Task as the future value.
var task = PerformTask(7);
// Do assorted work, then get result.
i = task.Result;
}