I want to retry when the NEST error is in a range of HttpCodeResponse and I have the follow generic policy:
public Policy<D> CreatePolicy<T, D>(
PolicyType policyType)
where T : Exception where D : IApiCallDetails
{
switch (policyType)
{
case PolicyType.WaitAndRetryAsync:
var httpStatusCodesWorthRetrying = new List<string>(this.policyConfiguration.HttpStatusCodesToRetrying.Split(','));
return Policy.Handle<T>()
.OrResult<D>(r => httpStatusCodesWorthRetrying.Select(int.Parse).ToList().Contains(r.HttpStatusCode.Value))
.WaitAndRetryAsync(
this.policyConfiguration.NumberOfRequestRetries,
retryAttempt => TimeSpan.FromSeconds(this.policyConfiguration.TimeInSecondsBetweenRetries),
onRetry: (exception, retryCount, context) =>
{
Log.Error($"[{context.PolicyKey}] Retry {retryCount} due to {exception.Exception.Message}.");
throw exception.Exception;
})
.WithPolicyKey(nameof(PolicyType.WaitAndRetryAsync));
default:
throw new ArgumentOutOfRangeException(nameof(policyType), policyType, null);
}
But when I try to apply to an elasticClient call, I receive the error:
can not implicity convert System.Threading.Tasks.Task<Nest.ISearchResponse<Model.Product>> to System.Threading.Tasks.Task<Elasticsearch.Net.IApiCallDetails>
Policy<IApiCallDetails> policyWaintAndRetry = this.policyFactory.CreatePolicy<ElasticsearchClientException, IApiCallDetails>(PolicyType.WaitAndRetryAsync);
var searchResponse = await policyWaintAndRetry.ExecuteAsync
action: () =>
this.client.SearchAsync<Product>(s => s
.From((request.PageNumber - 1) * request.PageSize)
.Size(request.PageSize)
.Index(GetIndexName(request.TenantId))
.Query(q => tq), CancellationToken.None),
contextData: new Polly.Context("SearchProductSearchAsync"))
.ConfigureAwait(false);
For NEST 5.x, I think D should have a generic parameter constraint of IResponse; every response within NEST implements IResponse and the ApiCall property inherited from IBodyWithApiCallDetails contains the IApiCallDetails with the HTTP status code.
It is not necessary to define two separate policies for handling exceptions and results. The two separate policies in this answer could be combined like this:
public Policy<TResult> CreatePolicyForResultAndException<TResult, TException>(PolicyType policyType)
where TResult : HttpResponseMessage
where TException: Exception
{
switch (policyType)
{
case PolicyType.WaitAndRetryAsync:
var httpStatusCodesWorthRetrying = new List<string>(this.policyConfiguration.HttpStatusCodesToRetrying.Split(','));
return Policy.HandleResult<TResult>(r => httpStatusCodesWorthRetrying.Select(int.Parse).ToList().Contains((int)r.StatusCode))
.Or<TException>()
.WaitAndRetryAsync(
this.policyConfiguration.NumberOfRequestRetries,
retryAttempt => TimeSpan.FromSeconds(this.policyConfiguration.TimeInSecondsBetweenRetries),
onRetry: (outcome, retryCount, context) =>
{
Log.Error($"[{context.PolicyKey}] Retry {retryCount} due to {outcome.Result ?? outcome.Exception.Message}.");
if (outcome.Exception != null) throw outcome.Exception; // [*] if desired - see note after code sample
})
.WithPolicyKey(nameof(PolicyType.WaitAndRetryAsync));
default:
throw new ArgumentOutOfRangeException(nameof(policyType), policyType, null);
}
}
[*] This line in the code sample above preserves the throwing of the exception within the onRetry from the original answer. However, it would be unusual to rethrow the exception within the onRetry, as the policy will not handle the exception rethrown there; throwing within onRetry will cause the policy to exit without making further tries.
Actually my problem was that I was trying to create a single policy for handling exception and result, while in fact I should create one for each and then merge them with a policyWrap
1st policy (for result):
public Policy<T> CreatePolicyForResult<T>(
PolicyType policyType)
where T : HttpResponseMessage
{
switch (policyType)
{
case PolicyType.WaitAndRetryAsync:
var httpStatusCodesWorthRetrying = new List<string>(this.policyConfiguration.HttpStatusCodesToRetrying.Split(','));
return Policy.HandleResult<T>(r => httpStatusCodesWorthRetrying.Select(int.Parse).ToList().Contains((int)r.StatusCode))
.WaitAndRetryAsync(
this.policyConfiguration.NumberOfRequestRetries,
retryAttempt => TimeSpan.FromSeconds(this.policyConfiguration.TimeInSecondsBetweenRetries))
.WithPolicyKey(nameof(PolicyType.WaitAndRetryAsync));
default:
throw new ArgumentOutOfRangeException(nameof(policyType), policyType, null);
}
}
2nd policy (for exception):
public Policy CreatePolicyForException<T>(
PolicyType policyType)
where T : Exception
{
switch (policyType)
{
case PolicyType.WaitAndRetryAsync:
return Policy.Handle<T>()
.WaitAndRetryAsync(
this.policyConfiguration.NumberOfRequestRetries,
retryAttempt => TimeSpan.FromSeconds(this.policyConfiguration.TimeInSecondsBetweenRetries),
onRetry: (exception, retryCount, context) =>
{
Log.Error($"[{context.PolicyKey}] Retry {retryCount} due to {exception.Message}.");
throw exception;
})
.WithPolicyKey(nameof(PolicyType.WaitAndRetryAsync));
default:
throw new ArgumentOutOfRangeException(nameof(policyType), policyType, null);
}
}
Usage:
var policyWaintAndRetryForExeption = this.policyFactory.CreatePolicyForException<ElasticsearchClientException>(PolicyType.WaitAndRetryAsync);
var policyWaintAndRetryForResult = this.policyFactory.CreatePolicyForResult<HttpResponseMessage>(PolicyType.WaitAndRetryAsync);
PolicyWrap<HttpResponseMessage> policyWrap = policyWaintAndRetryForExeption.WrapAsync(policyWaintAndRetryForResult);
Related
I have an API that has certain limits defined. Since I have used Polly C# library to limit the calls made to API. Below is the policy I am using.
var _rateLimitPolicy = Policy.RateLimitAsync(10,TimeSpan.FromSeconds(1), 5);
var _retryPolicy = Policy
.Handle<RateLimitRejectedException>(e => e.RetryAfter > TimeSpan.Zero)
.WaitAndRetryAsync(
retryCount: 3,
sleepDurationProvider: (i, e, ctx) =>
{
var rle = (RateLimitRejectedException)e;
return rle.RetryAfter;
},
onRetryAsync: (e, ts, i, ctx) => Task.CompletedTask
);
_wrappedPolicies = Policy.WrapAsync(_retryPolicy, _rateLimitPolicy);
Currently, once the retry limit of 3 is exceeded it throws RateLimitRejectedException. I want to throw a custom error if the retry limit is exceeded. Does anyone know how to do it?
If you want to throw an exception then use System.Exception with an if statement
So:
if(retryLimit>3)
{
throw new System.Exception("Retry Limit exceeded 3.");
}
If you don't wish to interrupt the process with a thrown exception, you can use a try{}catch(Exception e){} too
Whenever you want to execute your _wrappedPolicies
then you can call either ExecuteAsync or ExecuteAndCaptureAsync methods.
Former throws the original exception in case of retry
Latter captures the result in a PolicyResult both in failure and success cases
PolicyResult policyExecutionResult = await _wrappedPolicies.ExecuteAndCaptureAsync(...);
On this result object there is a property called FinalException. You can examine that and based on the assessment result you can throw a custom exception
if (policyExecutionResult.FinalExecption is RateLimitRejectedException)
throw new CustomException(...);
I came to know through Polly I can re-try the execution up to multiple configurable time like below example,
Policy
.Handle<SqlException>(ex => ex.Number == 1205)
.Or<ArgumentException>(ex => ex.ParamName == "example")
.WaitAndRetry(3, _ => TimeSpan.FromSeconds(3))
.Execute(DoSomething);
In my case below I am doing JSON serialization and all to find the error code 403002 and only on this error I want to re-try my code await deviceClient.SendEventAsync(new Message()); with a configuration time.
How to achieve this with Polly?
try
{
await deviceClient.SendEventAsync(new Message());
}
catch (Exception ex)
{
var errorMessage = ex.InnerException?.Message;
if (errorMessage != null)
{
dynamic error = JsonConvert.DeserializeObject(errorMessage);
if (error != null && error.errorCode == 403002)
{
// want to re-try again 'await deviceClient.SendEventAsync'
}
else
{
_logger.LogError($"Other error occurred : {error}");
}
}
}
finally
{
//
}
You can Handle the generic Exception and validate the errorCode to return bool value.
Policy
.Handle<Exception>(ex =>
{
var errorMessage = ex.InnerException?.Message;
if (errorMessage != null)
{
dynamic error = JsonConvert.DeserializeObject(errorMessage);
if (error != null && error.errorCode == 403002)
{
// want to re-try again 'await deviceClient.SendEventAsync'
return true;
}
}
return false;
})
.WaitAndRetry(3, _ => TimeSpan.FromSeconds(3))
.Execute(DoSomething);
I might be late to the party, but let me put my 2 cents here.
user1672994's suggested solution will not work for SendEventAsync, because the policy definition is created for sync function whereas your to-be-decorated one is async.
There is simple fix for that: use WaitAndRetryAsync instead.
I would also suggest to extract your should trigger retry delegate into a separate method to make your policy definition short and concise like this:
var retryPolicy = Policy
.Handle<Exception>(ShouldTriggerRetry)
.WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(3));
bool ShouldTriggerRetry(Exception ex)
{
var errorMessage = ex.InnerException?.Message;
if (string.IsNullOrEmpty(errorMessage)) return false;
var error = JsonConvert.DeserializeObject<dynamic>(errorMessage);
var shouldTrigger = error?.errorCode == 403002;
if(!shouldTrigger) _logger.LogError($"Other error occurred : {error}");
return shouldTrigger;
}
One final thought: As I can see in your code you have a finally block. If you need to do some cleanup between the retry attempts then you should do that inside onRetry or onRetryAsync delegate (depends on your cleanup code: sync or async).
var retryPolicy = Policy
.Handle<Exception>(ShouldTriggerRetry)
.WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(3), SyncCleanup);
//...
void SyncCleanup(Exception ex, TimeSpan sleep)
{
//...
}
var retryPolicy = Policy
.Handle<Exception>(ShouldTriggerRetry)
.WaitAndRetryAsync(3, _ => TimeSpan.FromSeconds(3), AsyncCleanup);
//...
Task AsyncCleanup(Exception ex, TimeSpan sleep)
{
//...
}
WaitAndRetryAsync has 19 overloads so, you might need to scrutinise them if you need to access for example the retry counter or the Context in your onRetry(Async) delegate.
UPDATE #1
Where should I put await deviceClient.SendEventAsync(new Message())
In case of Polly you define a policy and then you use it. The above code was just the definition. The usage looks like this:
await retryPolicy.ExecuteAsync(async () => await deviceClient.SendEventAsync(new Message()));
or like this
await retryPolicy.ExecuteAsync(() => deviceClient.SendEventAsync(new Message()));
Which block I should put my code plus after 3 fail I want to log message and initialize some variable
If all your attempts fail (4 in your case: the initial attempt + 3 retries) then the retry policy will throw the original exception
try
{
await retryPolicy.ExecuteAsync(async () => await deviceClient.SendEventAsync(new Message()));
}
catch(Exception ex)
{
_logger.LogError(ex, "Operation failed 4 times");
}
We have to implement a retry-mechanism.
To test the RetryProvider, I want a fake of a class to throw exceptions on the first two calls, but return a valid object on the third call.
Under normal circumstances (without throwing exceptions) we could use A.CallTo(() => this.fakeRepo.Get(1)).ReturnsNextFromSequence("a", "b", "c");
I want something similar:
First Call: throw new Exception();
Second Call: throw new Exception();
Third Call: return "success";
How can I configure my fake to do this?
Thanks in advance
var fakeRepo = A.Fake<IFakeRepo>();
A.CallTo(() => fakeRepo.Get(1))
.Throws<NullReferenceException>()
.Once()
.Then
.Throws<NullReferenceException>()
.Once()
.Then
.Returns('a');
See more about this at Specifying different behaviors for successive calls.
This should work:
A.CallTo(() => this.fakeRepo.Get(1))
.Throws<Exception>().Twice()
.Then
.Returns("a");
Another way do it like the sequence:
var funcs = new Queue<Func<string>>(new Func<string>[]
{
() => throw new Exception(),
() => throw new Exception(),
() => "a",
});
A.CallTo(() => this.fakeRepo.Get(1)).ReturnsLazily(() => funcs.Dequeue().Invoke()).NumberOfTimes(queue.Count);
Could have extension method:
public static IThenConfiguration<IReturnValueConfiguration<T>> ReturnsNextLazilyFromSequence<T>(
this IReturnValueConfiguration<T> configuration, params Func<T>[] valueProducers)
{
if (configuration == null) throw new ArgumentNullException(nameof(configuration));
if (valueProducers == null) throw new ArgumentNullException(nameof(valueProducers));
var queue = new Queue<Func<T>>(valueProducers);
return configuration.ReturnsLazily(x => queue.Dequeue().Invoke()).NumberOfTimes(queue.Count);
}
An call it like this:
A.CallTo(() => this.fakeRepo.Get(1)).ReturnsNextLazilyFromSequence(
() => throw new Exception(),
() => throw new Exception(),
() => "a");
I am using polly policy for retry attempts in the following way:
results = await Policy
.Handle<WebException>()
.WaitAndRetryAsync
(
retryCount: 5,
sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))
)
.ExecuteAsync(async () => await task.Invoke());
I am using the AsyncErrorHandler to handle all the web exceptions:
public static class AsyncErrorHandler
{
public static void HandleException(Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
However some of the expections I would like to throw up to the GUI.
With this code in place how can I prevent handling of a specific exception and instead throw it to the GUI?
[UPDATE] If I throw a specific exception inside the HandleException function I receive an Unhandled Error Message dialog in Visual Studio.
Implement it differently to only throw errors on the ones you want displayed to a user, then catch those you want to throw and do what you want with their contents (either show to user or not).
try
{
results = await Policy
.Handle<WebException>()
.WaitAndRetryAsync
(
retryCount: 5,
sleepDurationProvider: retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt))
)
.ExecuteAsync(async () => await task.Invoke());
}
catch (ExceptionToThrowToUser ex)
{
MessageBox.Show(ex.Message);
}
public static class AsyncErrorHandler
{
public static void HandleException(Exception ex)
{
if (ex is ExceptionToThrowToUser)
{
throw;
}
else
Debug.WriteLine(ex.Message);
}
}
Editted for update.
For help handling errors: Best practices for catching and re-throwing .NET exceptions
I am using Moq as my mocking framework and I need to test a class that when a specific type of exception is run it will keep trying until the situation is resolved once that happens the execution finishes.
So what I need is something similar to:
myMock = Mock<IFoo>();
myMock.Setup(m => m.Excecute()).Throws<SpecificException>();
myMock.Setup(m => m.Execute());
var classUnderTest = MyClass(myMock);
classUnderTest.DoSomething();
Assert.AreEqual(expected, classUnderTest.Result);
Thanks for any help you can give.
This is one way, based on the Moq QuickStart example of returning different values on each invocation.
var mock = new Mock<IFoo>();
var calls = 0;
mock.Setup(foo => foo.GetCountThing())
.Returns(() => calls)
.Callback(() =>
{
calls++;
if (calls == 1)
{
throw new InvalidOperationException("foo");
}
});
try
{
Console.WriteLine(mock.Object.GetCountThing());
}
catch (InvalidOperationException)
{
Console.WriteLine("Got exception");
}
Console.WriteLine(mock.Object.GetCountThing());
If the method returns void, use:
var myMock = new Mock<IFoo>();
bool firstTimeExecuteCalled = true;
myMock.Setup(m => m.Execute())
.Callback(() =>
{
if (firstTimeExecuteCalled)
{
firstTimeExecuteCalled = false;
throw new SpecificException();
}
});
try
{
myMock.Object.Execute();
}
catch (SpecificException)
{
// Would really want to call Assert.Throws instead of try..catch.
Console.WriteLine("Got exception");
}
myMock.Object.Execute();
Console.WriteLine("OK!");
https://github.com/Moq/moq4/wiki/Quickstart#miscellaneous
Setting up a member to return different values / throw exceptions on sequential calls:
var mock = new Mock<IFoo>();
mock.SetupSequence(f => f.GetCount())
.Returns(3) // will be returned on 1st invocation
.Returns(2) // will be returned on 2nd invocation
.Returns(1) // will be returned on 3rd invocation
.Returns(0) // will be returned on 4th invocation
.Throws(new InvalidOperationException()); // will be thrown on 5th invocation
Why not actually write your own test object that does this? If it's just going to be used for testing ie something like:
public class Mock : IFoo
{
private int _calls;
public Mock()
{
_calls = 0;
}
public int Execute()
{
_calls++;
if (_calls == 1)
throw new Exception();
return value;
}
}