I have a .NET Core worker service that fetches a list of procedures from DB that needs to execute on a regular time interval.
For every procedure Task is created with a delay after procedure executed. Different procedures have different delays.
protected override async Task ExecuteAsync(CancellationToken cancellationToken)
{
var proceduresToExec = await _sqlRepository.GetProceduresToExecute(cancellationToken);
var workers = new List<Task>();
foreach (var proc in proceduresToExec)
{
workers.Add(ExecuteProcedure(proc, cancellationToken));
}
// IMPORTANT to wait for all tasks
await Task.WhenAll(workers);
}
Method that executes procedure is:
private async Task ExecuteProcedure(ProcToExec proc, CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
try
{
// This line throws multiple exceptions per 1ms when SQL is unavailable
await _sqlRepository.ExecuteStoredProcedureAsync(proc, cancellationToken);
await Task.Delay(proc.Delay, cancellationToken);
}
catch (Exception ex)
{
// exceptions are logged here
_logger.LogError(ex);
}
}
}
SQL access is:
public async Task ExecuteStoredProcedureAsync(ProcToExec proc, CancellationToken cancellationToken)
{
var connStr = _configuration["DB_Conn_string"];
using (SqlConnection conn = new SqlConnection(connStr))
using (SqlCommand cmd = new SqlCommand($"dbo.{proc.ProcedureName}", conn))
{
await conn.OpenAsync(cancellationToken);
cmd.CommandType = CommandType.StoredProcedure;
await cmd.ExecuteNonQueryAsync(cancellationToken);
}
return;
}
Problem begins when SQL instance becomes unavailable.
Expected behaviour: Program tries to execute stored procedure, instance is unavailable. When timeout expires, throw error and then wait for delay (2000ms). After delay is done, retry.
Reality: When SQL instance becomes unavailable, logger logs multiple messages per 1ms. Average is ~8k logged errors per 1s.
Can anyone point me in direction what am I doing wrong?
See code comments:
private async Task ExecuteProcedure(ProcToExec proc, CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
try
{
// This line throws multiple exceptions per 1ms when SQL is unavailable
await _sqlRepository.ExecuteStoredProcedureAsync(proc, cancellationToken);
// If above line throws, this following line
// WILL NOT BE EXECUTED!
// Control flow will jump into the `catch` and go
// immediately into the next iteration.
await Task.Delay(proc.Delay, cancellationToken);
}
catch (Exception ex)
{
// exceptions are logged here
_logger.LogError(ex);
}
}
}
Also mind that this will only ever leave the loop IF cancellation is requested.
If you want to return on successful execution, you have to actually make it do that.
According to your requirement "Expected behaviour: Program tries to execute stored procedure, instance is unavailable. When timeout expires, throw error and then wait for delay (2000ms). After delay is done, retry." - I'd have expected to see something like this:
private async Task ExecuteProcedure(ProcToExec proc, CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
try
{
await _sqlRepository.ExecuteStoredProcedureAsync(proc, cancellationToken);
return;
}
catch (Exception ex)
{
// exceptions are logged here
_logger.LogError(ex);
await Task.Delay(proc.Delay, cancellationToken);
}
}
}
Maybe also consider Polly to introduce error handling policies (not affiliated to the project).
Another after-thought:
Even in the last snippet, that should somewhat do what's expected, you do not have a means to report to the caller if the procedure has been executed successfully or if the operation has been cancelled. Maybe think about whether you need that.
Related
The while loop currently blocks the thread as it retries forever until the connection is established. I expect it to retry forever but it shouldn't block the thread (just like if we call StartAsync without awaiting it) and make it possible for us to call StopAsync during StartAsync's execution and cancel that process of retrying forever. It would require passing a CancellationToken to ConnectAsync too.
await client.StartAsync(); // do not block
await Task.Delay(5000);
await client.StopAsync();
I was thinking of moving the CancellationToken before the while loop and pass it to the while loop as well as loop until while (!await ConnectAsync().ConfigureAwait(false) && !_tokenSource.IsCancellationRequested) and then wrap that logic into a Task.Run to prevent blocking. What do you think?
Full code (#GeneralClient2 class)
public async Task StartAsync()
{
// Prevent a race condition
await _semaphore.WaitAsync().ConfigureAwait(false);
try
{
if (IsRunning)
{
return;
}
while (!await ConnectAsync().ConfigureAwait(false))
{
}
IsRunning = true;
Debug.Assert(_clientWebSocket != null);
_tokenSource = new CancellationTokenSource();
_processingSend = ProcessSendAsync(_clientWebSocket, _tokenSource.Token);
_processingData = ProcessDataAsync(_tokenSource.Token);
_processingReceive = ProcessReceiveAsync(_clientWebSocket);
}
finally
{
_semaphore.Release();
}
}
public async Task StopAsync()
{
if (!IsRunning)
{
return;
}
_logger.LogDebug("Stopping");
try
{
if (_clientWebSocket is { State: not (WebSocketState.Aborted or WebSocketState.Closed or WebSocketState.CloseSent) })
{
await _clientWebSocket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, string.Empty, CancellationToken.None).ConfigureAwait(false);
}
}
catch
{
}
await _processingReceive.ConfigureAwait(false);
_logger.LogDebug("Stopped");
}
private async ValueTask<bool> ConnectAsync()
{
_logger.LogDebug("Connecting");
var ws = new ClientWebSocket();
try
{
await ws.ConnectAsync(new Uri(_url), CancellationToken.None).ConfigureAwait(false);
Connected?.Invoke(this, EventArgs.Empty);
}
catch (Exception) // WebSocketException or TaskCanceledException
{
ws.Dispose();
return false;
}
_clientWebSocket = ws;
_logger.LogDebug("Connected");
return true;
}
The while loop currently blocks the thread
Are you sure? because ClientWebSocket.ConnectAsync explicitly states
This operation will not block. The returned Task object will complete
after the connect request on the ClientWebSocket instance has
completed.
So ConnectAsync should not block, and therefore neither the while loop. But even if it is non-blocking it might still consume significant CPU usage. Or something might throw before you even call ClientWebSocket.ConnectAsync, and since you just eat all the exceptions you will never know.
make it possible for us to call StopAsync during StartAsync's execution and cancel that process of retrying forever
You should be careful when stopping some service in a threaded or async context. Since some other task might find that the needed resource have been disposed.
while (!await ConnectAsync().ConfigureAwait(false) && !_tokenSource.IsCancellationRequested)
The problem with this is that the loop will exit and continue running the method if no connection has been established. Because of this it is recommended to use ThrowIfCancellationRequested, even if it is somewhat painful to use exceptions for flow control.
My recommendations would be to make StartAsync take a cancellationToken that aborts the connection process. This method should return an object representing the connection, i.e. Task<MyConnection>. This connection object should be disposable. Stopping the connection could be done like
// Create connection
var cts = new CancellationTokenSource();
var myStartAsyncConnectionTask = StartAsync(cts.Token);
// Close connection
cts.Cancel();
try{
var myConnection = await myStartAsyncConnectionTask;
myConnection.Dispose();
}
catch(OperationCancelledException)
{
...
}
That should work regardless of what state the connection is in. If a connection has been established the cancellation will do nothing, and the object will be disposed. If it has failed to connect, awaiting the task should throw. Note that StartAsync need to be written so that it cleans up any created resource in case the method throws any exception at any stage.
I am trying to insert a record in azure table on some specific condition. What I found that the records are not getting saved all the time. Few times it works and few times its not. Here is the code which I have written to execute the insert. In addition to that, the code is written to insert a single record for each call and its never going to save multiple records at a time. Also I don't want to wait for the insert operation nor I want to do it synchronously. I want to do it asynchronously and that is without wait.
why the code sometimes fails to insert the records?
Here is my code snippet
try
{
AuditUtils utils = new AuditUtils((int)this.User.Id);
String tableName = "ServiceHeaderInfo", partitionKey = this.Header.PrimaryAccount;
TableAuditEntity tableAuditEntity = utils.GetAuditEntity(tableName, partitionKey, Guid.NewGuid().ToString(), TableAuditTypes.Insert, refId, refType);
tableAuditEntity["DeviceIdentifier"] = this.Header.DeviceId;
tableAuditEntity["DeviceModel"] = this.Header.DeviceModel;
tableAuditEntity["OSVersion"] = this.Header.OSVersion;
utils.AddAsync(tableAuditEntity).ContinueWith(t =>
{
var ex = t.Exception?.GetBaseException();
if (ex != null)
{
this.HandleError(ex);
}
}, System.Threading.Tasks.TaskContinuationOptions.OnlyOnFaulted);
}
catch (Exception ex)
{
this.HandleError(ex);
}
public async Task<TableResult> AddAsync(string tableName, SampleTableEntity entity, CancellationToken cancellationToken = default(CancellationToken))
{
var table = await GetTableAsync(tableName);
return await table.ExecuteAsync(TableOperation.InsertOrReplace(entity), cancellationToken);
}
It is not possible Async without Wait.
You need to use
ExecuteAsync(TableOperation)`
or
ExecuteAsync(TableOperation, TableRequestOptions, OperationContext) methods is used to achieve the requirement.
Instead of above methods you have
ExecuteAsync(TableOperation, CancellationToken)
&
ExecuteAsync(TableOperation, TableRequestOptions, OperationContext, CancellationToken)
When you use CancellationToken it must have a TaskFactory.ContinueWhenAll method.
In a cancellation of the operation, it instantiates a CancellationTokenSource object that generates a cancellation token that's passed to a TaskFactory object.
In turn, the TaskFactory object passes the cancellation token to each of the tasks responsible for collecting readings for a particular instrument.
The TaskFactory.ContinueWhenAll<TAntecedentResult, TResult>(Task<TAntecedentResult>[], Func<Task<TAntecedentResult>[],TResult>, CancellationToken) method is called to ensure that computed only after all readings have been gathered successfully.
If a task has not completed because it was cancelled, the TaskFactory.ContinueWhenAll method throws an exception.
Please refer: click here
I am trying to make a generic method that will cancel an Async web request and any additional operations associated with it. I found another question regarding this such as this.
And I have written a helper class that will do just this. I present it below:
public static class Helpers
{
public static async Task<string> GetJson(string url,
CancellationTokenSource cancellationTokenSource,
bool useSynchronizationContext = true)
{
try
{
var request = (HttpWebRequest)WebRequest.Create(url);
string jsonStringResult;
using (WebResponse response = await request.GetResponseAsync()
.WithCancellation(cancellationTokenSource.Token,
request.Abort, useSynchronizationContext))
{
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
jsonStringResult = await reader.ReadToEndAsync();
reader.Close();
dataStream.Close();
}
cancellationTokenSource.Token.ThrowIfCancellationRequested();
return jsonStringResult;
}
catch (Exception ex) when (ex is OperationCanceledException
|| ex is TaskCanceledException)
{
}
catch (Exception ex) when (ex is WebException
&& ((WebException)ex).Status == WebExceptionStatus.RequestCanceled)
{
}
catch (Exception ex)
{
//Any other exception
}
finally
{
cancellationTokenSource.Dispose();
}
return default;
}
public static async Task<T> WithCancellation<T>(this Task<T> task,
CancellationToken cancellationToken, Action action,
bool useSynchronizationContext)
{
using (cancellationToken.Register(action, useSynchronizationContext))
{
return await task;
}
}
}
Notice the line
cancellationTokenSource.Token.ThrowIfCancellationRequested();
right before returning the JSON string.
When the operation is canceled and the flow of execution is on line
StreamReader reader = new StreamReader(dataStream);
for example, all lines below (reader.Close() etc.) will be executed and the exception will be thrown when ThrowIfCancelationRequested() is executed - is that correct? Am I missing something?
If so, is there a way to cancel everything at once?
Thanks everyone for their response,
After the answer provided and all really useful comments I updated the implementation.I used HttpClient and the extension method in the link for having a task that cannot actually being canceled to behave like one that can - readAsStringAsync is actually executed since it cannot accept a cancellation token.
public static class Helpers
{
public static async Task<string> GetJson(string url,CancellationToken cancellationToken)
{
try
{
string jsonStringResult;
using (var client = new HttpClient())
{
cancellationToken.ThrowIfCancellationRequested();
using (var response = await client.GetAsync(url, cancellationToken))
{
jsonStringResult = await response.Content.ReadAsStringAsync().WithCancellation(cancellationToken);
}
}
return jsonStringResult;
}
catch (Exception ex) when (ex is OperationCanceledException)
{
}
catch (Exception ex) when (ex is WebException exception && exception.Status == WebExceptionStatus.RequestCanceled)
{
}
catch (Exception ex)
{
//LogException(ex);
}
return default;
}
public static Task<T> WithCancellation<T>(this Task<T> task, CancellationToken cancellationToken)
{
return task.IsCompleted
? task: task.ContinueWith(completedTask => completedTask.GetAwaiter().GetResult(),cancellationToken,TaskContinuationOptions.ExecuteSynchronously,TaskScheduler.Default);
}
}
all lines below (reader.Close() etc.) will be executed and the
exception will be thrown when ThrowIfCancelationRequested() is
executed - is that correct? Am I missing something?
If so, is there a way to cancel everything at once?
First of all, operation that you want to cancel has to explicitly support being canceled. So, you have to push your best to use a code for executing some operation in a way which somehow takes CancellationToken as an argument. If it is not possible, then you have no other chance.
That's why this concept is called Cooperative cancellation. Because almost always both sides should be aware of that cancellation happened. Client-side should be aware of that the code was actually canceled, it’s not enough for a client to know that cancellation was just requested. For the callee, it’s important to know about the fact that cancellation was requested in order to properly finish itself.
Regarding to checking whether operation is cancelled while executing Close method of stream and reader. You have to call cleanup methods always whether operation is cancelled or not if you want to avoid memory leaks. Of course I suggest you to use using statement which will automatically do that cleanup.
And by the way, for making some function cancelable you don't have to check whether cancellation is requested before executing each line. You just have to check whether cancellation is request before and after executing some long-running operation. And pass cancellation token if that long-running operations supports cancelling through cancellation token property.
Also, you have to take a look for side-effects. Don’t cancel if you’ve already incurred side-effects that your method isn’t prepared to revert on the way out that would leave you in an inconsistent state.
Some general code-block might be so:
if(ct.IsCancellationRequested)
{
break; // or throw
}
await DoSomething(ct);
if (ct.IsCancellationRequested)
{
// if there is no side-effect
return; // or throw
// or, we already did something in `DoSomething` method
// do some rollback
}
As a solution, you can use some different objects like HttpClient or WebRequest for executing async, awaitable and cancelable web request. You can take a look to that link for implementation details.
I have some code that is validating some data by making calls to a number of other services. I start all of the calls in parallel and then wait until at least one of them finishes. If any of the requests fail, I don't care about the result of the other calls.
I make the calls with HttpClient and I have passed an HttpMessageHandler in that does a bunch of logging. Essentially:
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
HttpResponseMessage response = null;
try
{
response = await base.SendAsync(request, cancellationToken);
}
catch (OperationCanceledException ex)
{
LogTimeout(...);
throw;
}
catch (Exception ex)
{
LogFailure(...);
throw;
}
finally
{
LogComplete(...);
}
return response;
}
No the part that I'm having trouble with is when I cancel the requests. When I cancel a request, I'm doing it on purpose, so I don't want it to get logged as a timeout, but there doesn't appear to be any difference between a cancellation and a real timeout.
Is there anyway to accomplish this?
Edit: I need to clarify this, a little bit. The service making the calls in parallel is passing in CancellationTokens with a timeout:
var ct = new CancellationTokenSource(TimeSpan.FromSeconds(2));
So when the server takes more than two seconds to respond, I get an OperationCanceledException, and if I manually cancel the token source (say because another server returned an error after 1 second), then I still get an OperationCanceledException. Ideally, I would be able to look at CancellationToken.IsCancellationRequested to determine if it was cancelled due to a timeout, as opposed to explicitly requested to be cancelled, but it appears that you get the same value regardless of how it was canceled.
If you want to distinguish the two cancellation types, then you need to use two different cancellation tokens. There's no other way. This is not too hard since they can be linked - just a bit awkward.
The cleanest way to write this IMO is to move the timeout code into the SendAsync method instead of the calling method:
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
using (var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken))
{
cts.CancelAfter(TimeSpan.FromSeconds(2));
try
{
return await base.SendAsync(request, cts.Token);
}
catch (OperationCanceledException ex)
{
if (cancellationToken.IsCancellationRequested)
return null;
LogTimeout(...);
throw;
}
catch (Exception ex)
{
LogFailure(...);
throw;
}
finally
{
LogComplete(...);
}
}
}
If you don't want to move the timeout code into SendAsync, then you'll need to do the logging outside of that method, too.
If the exceptions aren't telling you the difference between the two cases then you will need to check with either the Task or the CancellationToken to see if there was actually a cancellation.
I would lean toward asking the Task which will have its IsCanceled property return true if an unhandled OperationCanceledException was thrown (using CancellationToken.ThrowIfCancellationRequested inside base.SendAsync most likely). Something like this...
HttpResponseMessage response = null;
Task sendTask = null;
try
{
sendTask = base.SendAsync(request, cancellationToken);
await sendTask;
}
catch (OperationCanceledException ex)
{
if (!sendTask.IsCancelled)
{
LogTimeout(...);
throw;
}
}
EDIT
In response to the update to the question, I wanted to update my answer. You are right cancellation whether it is specifically requested on the CancellationTokenSource or if it is caused by a timeout will lead to exactly the same outcome. If you decompile CancellationTokenSource you will see that for the timeout it just sets a Timer callback that will explicitly call CancellationTokenSource.Cancel when the timeout is reached, so both ways will end up calling the same Cancel method.
I think if you want to tell the difference you will need to derive from CancellationTokenSource (it isn't a sealed class) and then add your own custom cancel method that will set a flag to let you know that you explicitly cancelled the operation rather than letting it time out.
This is unfortunate since you will have both your custom cancel method and the original Cancel method available and will have to be sure to use the custom one. You may be able to get away with your custom logic just hiding the existing Cancel operation with something like this:
class CustomCancellationTokenSource : CancellationTokenSource
{
public bool WasManuallyCancelled {get; private set;}
public new void Cancel()
{
WasManuallyCancelled = true;
base.Cancel();
}
}
I would think that hiding the base method will work, you can give it a shot and find out.
I am starting a HubConnection. First I get a new Instance of HubConnection,
afterwards I create a new IHubProxy with Name = "fileHub" so far so good.
My problem is located at (or in) the awaitable Function ContinueWith, I try to start the Connection, and write to the console wheater the start was successful or not.
The connection.Start() is successful and "Connected" is written to the console.
Code added after the Console.WriteLine("Connected") is executed without problems, too.
But the Task never finishes, so the Client Class calling the HandleSignalRAsync() Method waits for the completion unsuccessfully.
Adding a return; or task.dispose(); does not solve my issue.
public static async Task HandleSignalRAsync()
{
connection = new HubConnection("http://localhost:12754");
myHub = connection.CreateHubProxy("fileHub");
await connection.Start().ContinueWith(
task =>
{
if (task.IsFaulted)
{
var ex = task.Exception.GetBaseException();
Console.WriteLine("There was an error opening the connection: {0}", ex);
}
else
{
Console.WriteLine("Connected");
}
});
}
I call the Method with the TaskAwaiter in another Class of my Solution:
Functions.HandleSignalRAsync().GetAwaiter().GetResult();
calling it with:
Functions.HandleSignalRAsync().Wait();
does not work, too.
But the Task never finishes
Because both examples synchronously block, causing your code to deadlock. You shouldn't be blocking on async code.
You'll need to properly asynchronously wait on HandleSignalRAsync:
await Functions.HandleSignalRAsync().ConfigureAwait(false);
If you're already using async-await, there's no advantage to use a continuation style with ContinueWith, you can simply wrap the execution in a try-catch statement and await inside:
try
{
await connection.Start().ConfigureAwait(false);
Console.WriteLine("Connected");
}
catch (Exception e)
{
Console.WriteLine("There was an error opening the connection: {0}", e);
}