Develop Reference

Is it possible to unit test CallActivityWithRetryAsync and reproduce retries?

I have read the Microsoft Documentation but it doesn't mention CallActivityWithRetryAsync.
My IDurableOrchestrationContext is mocked for the call I'm making:
mockContext.Setup(c => c.CallActivityWithRetryAsync(nameof(SerialiseXml), It.IsAny<RetryOptions>(),
It.IsAny<InboundOrchestrationData>()))
.ThrowsAsync(new IOException());
and I can put a breakpoint at the calling point:
try
{
await context.CallActivityWithRetryAsync("SerialiseXml",
new RetryOptions(TimeSpan.FromMinutes(1), 3), data); // breakpoint
}
catch (Exception e)
{
log.LogError("Problem serialising xml.");
}
but my code only ever breaks here once when I would expect it to break three times according to the RetryOptions.
If I pass in a Mock<ILogger<Class>> and check the invocations there is also only one. I have also configured my mock using SetupSequence() and multiple .ThrowsAsync.

Do mind that if you start mocking
mockContext.Setup(c => c.CallActivityWithRetryAsync(nameof(SerialiseXml), It.IsAny<RetryOptions>(),
It.IsAny<InboundOrchestrationData>()))
.ThrowsAsync(new IOException());
there won't be any retries performed at all since it is mocked. There is no real implementation that is executed.
Inside the implementation of CallActivityWithRetryAsync there will be a retry of the specified activity call in case of a retryable failure. Only after all retry attempts have failed an exception will bubble up to your code. So I ever expect just one call to CallActivityWithRetryAsync.
Your exception handling code will also be just called once, and only when all retries have failed or there is another reason for the call to fail.
The implementation of CallActivityWithRetryAsync will look like this (simplified):
async Task<int> CallActivityWithRetryAsync(string activity, RetryOptions retryOptions)
{
int attempts = 1;
while (true)
{
try
{
return await CallActivity(activity);
}
catch
{
++attempts;
if (attempts > retryOptions.RetryCount)
{
throw;
}
}
}
}
Given this pseudocode you can easily see why the breakpoint hits only one.

How to throw a custom exception in polly rate limit exceeds the rate?

I have an API that has certain limits defined. Since I have used Polly C# library to limit the calls made to API. Below is the policy I am using.
var _rateLimitPolicy = Policy.RateLimitAsync(10,TimeSpan.FromSeconds(1), 5);
var _retryPolicy = Policy
.Handle<RateLimitRejectedException>(e => e.RetryAfter > TimeSpan.Zero)
.WaitAndRetryAsync(
retryCount: 3,
sleepDurationProvider: (i, e, ctx) =>
{
var rle = (RateLimitRejectedException)e;
return rle.RetryAfter;
},
onRetryAsync: (e, ts, i, ctx) => Task.CompletedTask
);
_wrappedPolicies = Policy.WrapAsync(_retryPolicy, _rateLimitPolicy);
Currently, once the retry limit of 3 is exceeded it throws RateLimitRejectedException. I want to throw a custom error if the retry limit is exceeded. Does anyone know how to do it?

If you want to throw an exception then use System.Exception with an if statement
So:
if(retryLimit>3)
{
throw new System.Exception("Retry Limit exceeded 3.");
}
If you don't wish to interrupt the process with a thrown exception, you can use a try{}catch(Exception e){} too

Whenever you want to execute your _wrappedPolicies
then you can call either ExecuteAsync or ExecuteAndCaptureAsync methods.
Former throws the original exception in case of retry
Latter captures the result in a PolicyResult both in failure and success cases
PolicyResult policyExecutionResult = await _wrappedPolicies.ExecuteAndCaptureAsync(...);
On this result object there is a property called FinalException. You can examine that and based on the assessment result you can throw a custom exception
if (policyExecutionResult.FinalExecption is RateLimitRejectedException)
throw new CustomException(...);

How to throw final error using WaitAndRetry and Execute?

I'm, trying to check a simple WaitAndRetry of Polly
class Program
{
public static void Main()
{
int i = 0;
var _retryPolicy = Policy
.Handle<Exception>()
.WaitAndRetry(Backoff.ExponentialBackoff(TimeSpan.FromSeconds(2), 10),
(exception, timespan) =>
{
Console.WriteLine($"Retry: {timespan}. \n ex: {exception}");
});
_retryPolicy.Execute(() =>
{
Console.WriteLine(i);
i++;
int.Parse("something");
});
Console.ReadLine();
}
}
And I want to throw a final exception after all the retries are failed. How can I do it?
Excepted Result:
Retry: ..
Retry: ..
Retry: ..
My new final error!
Thank You!

You don't have to explicitly re-throw the last exception.
The retry works in the following way:
If an exception is thrown by the decorated method and there is a related .Handle<TEx> or .Or<TEx> clause inside the policy definition then it checks if the retry limit has been reached or not
If the retry limit has not been exceeded then it will perform yet another retry attempt
If the retry limit has been reached then it will throw the last exception
If there is not related .Handle<TEx> or .Or<TEx> clause then it will throw the last exception
Here is a diagram from the official documentation
Please also note that if you run your application in a debug mode then the IDE might stop each time when the decorated method throws exception. It depends on your IDE settings.

Set durationOfBreak in Polly CircuitBreaker

I am using Polly Circuit Breaker for an API which throws a user defined Exception if service is down for maintenance.
The Exception will contain a value how long the service will be down.
Is it possible to configure/update the circuit breaker policy after first user exception is handled ?
e.g.
CircuitBreakerPolicy breaker = Policy.Handle<UserException>()
.CircuitBreaker(
exceptionsAllowedBeforeBreaking: 1,
durationOfBreak: TimeSpan.FromMinutes(1));
Example()
{
try
{
string response = await breaker.ExecuteAsync<String>(() =>
{
return client.GetStringAsync("/api/values/");
});
}
catch (UserException ex)
{
var downtime = GetDowntime(ex);
//how to update the duration of break ?
breaker.durationOfBreak = downtime;
}
catch (Exception ex)
{
…
}
}

It is not possible to change the durationOfBreak on an existing CircuitBreakerPolicy instance, after creation.
For general dynamic reconfiguration during running, the Polly team recommends atomically updating a policy instance stored in the PolicyRegistry.
For the specific scenario:
[the] API throws a user defined Exception if service is down for maintenance [which] will contain a value how long the service will be down
you could, more simply, use the circuit-breaker's manual controls:
.Isolate() to manually force the circuit open;
.Reset() to manually reset the circuit.
You could .Isolate() the circuit when the given exception is thrown; and set a Timer to call Reset() on it at the time that the exception indicates the service should be available again.

RX - rethrow an error in containing method

I need to translate an error in an RX stream (IObservable) into an exception in the method that contains the subscription to the stream
(because of this issue https://github.com/aspnet/SignalR/pull/1331 , Whereby errors arent serialised to clients.) Once this issue is fixed I will revert to handling error properly
e.g.
I have the following method
public IObservable<StreamItem> LiveStream()
{
_mySvc.Start();
return _mySvc.ThingChanged();
}
So I have tried to subscribe to the stream and rethrow the error, but it still doesnt get transmitted to the client:
public IObservable<StreamItem> LiveStream()
{
_mySvc.Start();
_mySvc.ThingChanged().Subscribe(item => {}, OnError, () => {});
return _mySvc.ThingChanged();
}
private void OnError(Exception exception)
{
throw new Exception(exception.Message);
}
What I need is the equivelent of throwing in the LiveStream method
e.g. this error is propogated to the client
public IObservable<StreamItem> LiveStream()
{
_mySvc.Start();
throw new Exception("some error message");
return _mySvc.ThingChanged();
}
any ideas how to achieve this?

I have found this as well, especially with a "contained" reactive pipeline—that is, one with a well-defined beginning and end. In situations like those, it may suffice to simply allow underlying exceptions to bubble up to the containing scope. But as you have found, that concept is rather foreign to Rx generally: what happens in the pipeline stays in the pipeline.
The only way out of this that I have found in a contained scenario is to "slip" the error out of the stream using Catch(), and hand back an empty IObservable to allow the stream to halt naturally (otherwise, you'll hang if you're awaiting an IObservable for completion).
This will not work within your LiveStream() method, because that context/scope should have passed out of existence long before you're consuming your stream. So, this will have to happen in the context that contains the whole pipeline.
Exception error = null;
var source = LiveStream()
.Catch<WhatYoureStreaming, Exception>(ex => {error = ex; return Observable.Empty<WhatYoureStreaming>(); })
...
await source; // if this is how you're awaiting completion
// not a real exception type, use your own
if (error != null) throw new ContainingException("oops", error);
Just don't throw error there at the end, you'll lose the original stack trace.

Try this code:
public IObservable<StreamItem> LiveStream()
{
_mySvc.Start();
return
_mySvc
.ThingChanged()
.Materialize()
.Do(x =>
{
if (x.Kind == NotificationKind.OnError)
{
OnError(x.Exception);
}
})
.Dematerialize();
}
I'm not sure that this is the best way to go - throwing exceptions like this can cause you grief inside a stream where you end up with the wrong exception handlers firing. You might need to find another solution.