I'm learning Rx and I wonder how I could print out the retries to the console, e.g. "Retry #1", "Retry #2", etc. I have seen some people using .Dump extension method that I cannot find.
using System.Reactive.Concurrency;
using System.Reactive.Linq;
using System.Reactive.Threading.Tasks;
var test = new Test(new HttpClient());
var result = await test.GetAsync();
Console.WriteLine($"Result: {result}");
public sealed class Test
{
private readonly HttpClient _httpClient;
public Test(HttpClient httpClient)
{
_httpClient = httpClient;
}
public Task<string> GetAsync()
{
return Observable
.FromAsync(() => _httpClient.GetAsync("http://www.mocky.io/v2/5e307edf3200005d00858b49"))
.SubscribeOn(TaskPoolScheduler.Default)
.Retry(5)
.Timeout(TimeSpan.FromSeconds(5))
.Do(x => Console.WriteLine($"Is message successful? {x.IsSuccessStatusCode}"))
.SelectMany(async x =>
{
var response = await x.Content.ReadAsStringAsync();
return response;
})
.Catch<string, TimeoutException>(_ => Observable.Return("Timeout"))
.Catch<string, Exception>(ex => Observable.Return(ex.Message))
.ToTask();
}
}
Here's a basic example of writing out retries:
Observable
.Defer<int>(() =>
{
int counter = 0;
return
Observable
.Defer<int>(() =>
{
Console.WriteLine(counter++ > 0 ? $"Retry #{counter - 1}" : "First");
return Notification.CreateOnError<int>(new Exception()).ToObservable();
})
.Retry(5);
})
.Subscribe(
x => Console.WriteLine(x),
e => Console.WriteLine(e.Message));
This outputs:
First
Retry #1
Retry #2
Retry #3
Retry #4
Exception of type 'System.Exception' was thrown.
The observable you're retrying should be wrapped in a Observable.Defer to prevent the potential for repeated values. And, whenever I use "external" state in an observable I also wrap it in a Observable.Defer.
Related
I am trying to create a combined resilience strategy for my Rest clients and for this I have written the following :
private static Policy circuitBreakerPolicy = Policy
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
And I am getting the error
Cannot implicitly convert type Polly.CircuitBreaker.AsyncCircuitBreaker to Polly.Policy
What am I missing ? I have checked a number of references online but I cannot come across a concise enough explanation of what causes this issue.
Usage :
Eventually I want to combine the CircuitBreaker policy above with a WaitandRetry policy but I cannot figure out how to extract the combined policy :
(This is working)
public static IAsyncPolicy<HttpResponseMessage> CreateResiliencePolicy()
{
var timeoutPolicy = Policy.TimeoutAsync(TimeSpan.FromSeconds(180));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
(result, timeSpan, context) =>
{
});
return timeoutPolicy.WrapAsync(waitAndRetryPolicy);
}
This is what I want (Not working) :
public static IAsyncPolicy<HttpResponseMessage> CreateResiliencePolicy()
{
var circuitBreakerPolicy = Policy
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
(result, timeSpan, context) =>
{
});
return circuitBreakerPolicy.WrapAsync(waitAndRetryPolicy);
//I want to do this instead
//public static Policy resilientAsyncStrategy =circuitBreakerPolicy.WrapAsync(waitAndRetryPolicy);
//Then return resilientAsyncStrategy
//return resilientAsyncStrategy;
}
And then use the returned policy instance as :
public async Task<IEnumerable<PaymentDetailDto>> GetAsync()
{
var items = await resilientAsyncStrategy.ExecuteAsync(async () => await client.GetAsync());
return items;
}
Polly defines the following four abstract Policy types:
Policy
Policy < TResult >
AsyncPolicy
Async < TResult >
Method
Function
Sync
Policy
Policy<TResult>
Async
AsyncPolicy
AsyncPolicy<TResult>
So, in your case the circuitBreakerPolicy should be defined like this:
private static AsyncPolicy circuitBreakerPolicy = Policy
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
Whenever you want to combine / chain two (or more) policies then you should consider to use PolicyWrap (reference). Please bear in mind that policy chain works in an escalation way which means if the inner policy can't handle the problem then it will propagate that to the next outer policy.
Please also bear in mind that policies should be compatible with each other. So, if one of them is async then other one should be as well. If the inner returns something then the outer should do the same.
So, your CreateResiliencePolicy could look like this:
public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
var circuitBreakerPolicy = Policy<HttpResponseMessage>
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
(result, timeSpan, context) =>
{
});
return Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy);
}
Please also bear in mind that ordering matters:
Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy) you have an inner Retry and an outer CB
Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicy) you have an inner CB and an outer Retry
The following two lines are equivalent:
circuitBreakerPolicy.WrapAsync(waitAndRetryPolicy);
Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy)
If you want to change the ordering of the policies then your strategy will work in a different way. If you want to use the CB as inner policy and the retry as outer then you should amend the waitAndRetryPolicy to handle BrokenCircuitException as well.
Couple of months ago I have put together a sample application which demonstrates how can you incrementally design your resilience strategy.
UPDATE: Add sample code
Testing the retry logic
I've used the following console app to test your retry policy:
private static HttpClient client = new HttpClient();
public static async Task Main(string[] args)
{
var strategy = CreateResilienceStrategy();
await strategy.ExecuteAsync(async (ct) =>
await client.GetAsync("https://httpstat.us/500", ct)
, CancellationToken.None);
Console.WriteLine("Finished");
}
public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
var circuitBreakerPolicy = Policy<HttpResponseMessage>
.Handle<TimeoutException>()
.CircuitBreakerAsync(3, TimeSpan.FromSeconds(2),
onBreak: (_, __) => Console.WriteLine("Break"),
onReset: () => Console.WriteLine("Reset"),
onHalfOpen: () => Console.WriteLine("HalfOpen"));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
onRetryAsync: (_, ts, ___) =>
{
Console.WriteLine($"Retry, penalty: {ts.Seconds} secs");
return Task.CompletedTask;
});
return Policy.WrapAsync(circuitBreakerPolicy, waitAndRetryPolicy);
}
The output:
Retry, penalty: 3 secs
Retry, penalty: 9 secs
Retry, penalty: 27 secs
Finished
As you can see,
it had performed all retries without luck
it did not trigger the circuit breaker
strategy.ExecuteAsync returned a response where the status code is 500
it continued the work after the Http call
Testing the circuit breaker logic
Because the CB policy was setup to trigger for TimeoutException that's why I set the client.Timeout to 1 millisecond in the Main. With that the application crashes with TaskCanceledException.
It does that because HttpClient throws TaskCanceledException in case of Timeout instead of TimeoutException. If you would use polly's Timeout policy then you would receive TimeoutRejectedException.
So, here is my amended to code to be able to test CB
private static HttpClient client = new HttpClient();
public static async Task Main(string[] args)
{
client.Timeout = TimeSpan.FromMilliseconds(1);
var strategy = CreateResilienceStrategy();
try
{
await strategy.ExecuteAsync(async (ct) =>
await client.GetAsync("https://httpstat.us/500", ct)
, CancellationToken.None);
Console.WriteLine("Finished");
}
catch (Exception ex)
{
Console.WriteLine("Failed with " + ex.GetType().Name);
}
}
public static AsyncPolicy<HttpResponseMessage> CreateResilienceStrategy()
{
var circuitBreakerPolicy = Policy<HttpResponseMessage>
.Handle<OperationCanceledException>()
.Or<TimeoutRejectedException>()
.CircuitBreakerAsync(1, TimeSpan.FromSeconds(2),
onBreak: (_, __) => Console.WriteLine("Break"),
onReset: () => Console.WriteLine("Reset"),
onHalfOpen: () => Console.WriteLine("HalfOpen"));
var waitAndRetryPolicy = Polly.Policy
.Handle<HttpRequestException>()
.Or<OperationCanceledException>()
.OrResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.InternalServerError)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)),
onRetryAsync: (_, ts, ___) =>
{
Console.WriteLine($"Retry, penalty: {ts.Seconds} secs");
return Task.CompletedTask;
});
return Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicy);
}
Modifications:
Set the timeout to 1 milliseconds
Changed the consecutive failed count of CB from 3 to 1
Changed the chaining: retry outer, cb inner
Added OperationCanceledException conditions for both policies
The output
Break
Retry, penalty: 3 secs
HalfOpen
Break
Retry, penalty: 9 secs
HalfOpen
Break
Retry, penalty: 27 secs
HalfOpen
Break
Failed with OperationCanceledException
This question already has answers here:
Ignore the Tasks throwing Exceptions at Task.WhenAll and get only the completed results
(3 answers)
Closed 1 year ago.
Given the following:
var tPass1 = Task.FromResult(1);
var tFail1 = Task.FromException<int>(new ArgumentException("fail1"));
var tFail2 = Task.FromException<int>(new ArgumentException("fail2"));
var task = Task.WhenAll(tPass1, tFail1, tFail2);
task.Wait();
the call to task.Wait() throws an AggregateException, whose inner exceptions contain the fail1 and fail2 exceptions. But how can I access the tPass1 successful result?
Is this possible?
I'm aware that I can get the result from the individual task after the WhenAll has finished, via tPass1.Result however is there a way to get them in an array to avoid having to manually track all the things feeding into the WhenAll?
Maybe
public async Task<Task[]> RejectFailedFrom(params Task[] tasks)
{
try
{
await Task.WhenAll(tasks);
}
catch(Exception exception)
{
// Handle failed tasks maybe
}
return tasks.Where(task => task.Status == TaskStatus.RanToCompletion).ToArray();
}
Usage
var tasks = new[]
{
Task.FromResult(1),
Task.FromException<int>(new ArgumentException("fail1")),
Task.FromException<int>(new ArgumentException("fail2"))
};
var succeed = await RejectFailedFrom(tasks);
// [ tasks[0] ]
When a task fails we cannot access its Result property because it throws. So to have the results of a partially successful WhenAll task, we must ensure that the task will complete successfully. The problem then becomes what to do with the exceptions of the failed internal tasks. Swallowing them is probably not a good idea. At least we would like to log them. Here is an implementation of an alternative WhenAll that never throws, but returns both the results and the exceptions in a ValueTuple struct.
public static Task<(T[] Results, Exception[] Exceptions)> WhenAllEx<T>(
params Task<T>[] tasks)
{
ArgumentNullException.ThrowIfNull(tasks);
tasks = tasks.ToArray(); // Defensive copy
return Task.WhenAll(tasks).ContinueWith(t => // return a continuation of WhenAll
{
T[] results = tasks
.Where(t => t.IsCompletedSuccessfully)
.Select(t => t.Result)
.ToArray();
AggregateException[] aggregateExceptions = tasks
.Where(t => t.IsFaulted)
.Select(t => t.Exception) // The Exception is of type AggregateException
.ToArray();
Exception[] exceptions = new AggregateException(aggregateExceptions).Flatten()
.InnerExceptions.ToArray(); // Flatten the hierarchy of AggregateExceptions
if (exceptions.Length == 0 && t.IsCanceled)
{
// No exceptions and at least one task was canceled
exceptions = new[] { new TaskCanceledException(t) };
}
return (results, exceptions);
}, default, TaskContinuationOptions.DenyChildAttach |
TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
Usage example:
var tPass1 = Task.FromResult(1);
var tFail1 = Task.FromException<int>(new ArgumentException("fail1"));
var tFail2 = Task.FromException<int>(new ArgumentException("fail2"));
var task = WhenAllEx(tPass1, tFail1, tFail2);
task.Wait();
Console.WriteLine($"Status: {task.Status}");
Console.WriteLine($"Results: {String.Join(", ", task.Result.Results)}");
Console.WriteLine($"Exceptions: {String.Join(", ", task.Result.Exceptions.Select(ex => ex.Message))}");
Output:
Status: RanToCompletion
Results: 1
Exceptions: fail1, fail2
Playing around with #Theodor Zoulias's powerfull and elegant solution pushed me to something. It looks hacky, but still works. One can continue Task.WhenAll with something that will not throw an exception for sure (e.g. _ => { }) and Wait that something.
var cts = new CancellationTokenSource();
cts.Cancel();
var canceled = Task.Run(() => 1, cts.Token);
var faulted = Task.FromException<int>(new Exception("Some Exception"));
var ranToCompletion = Task.FromResult(1);
var allTasks = new[] { canceled, faulted, ranToCompletion };
// wait all tasks to complete regardless anything
Task.WhenAll(allTasks).ContinueWith(_ => { }).Wait();
foreach(var t in allTasks)
{
Console.WriteLine($"Task #{t.Id} {t.Status}");
if (t.Status == TaskStatus.Faulted)
foreach (var e in t.Exception.InnerExceptions)
Console.WriteLine($"\t{e.Message}");
if (t.Status == TaskStatus.RanToCompletion)
Console.WriteLine($"\tResult: {t.Result}");
}
Output looks like this:
Task #2 Canceled
Task #1 Faulted
Some Exception
Task #5 RanToCompletion
Result: 1
ConcurrentQueue<Exception> errorList = new();
var someTasks = makeSomeActionAsync().ContinueWith(x =>
{
if(x.Exception !=null)
errorList.Enqueue(x.Exception);
});
await Task.WhenAll(someTasks);
if(errorList.Any())
throw new Exception($"\n{string.Join("\n", errorList )}");
Change
var task = Task.WhenAll(tPass1, tFail1, tFail2);
task.Wait();
to
var all = new Task<int>[] { tPass1, tFail1, tFail2 }
.Where(t => t.Status == TaskStatus.RanToCompletion);
var task = Task.WhenAll(all);
task.Wait();
Working example
Let's say I have a list of objects myObjs of type List<Foo>.
I have a polly policy :
var policy = Policy.Handle<Exception>().RetryForever();
which i want to run methods on in parralel, but keep retrying each as they fail.
for (int i = 0; i < myObjs.Count; i++)
{
var obj = myObjs[i];
policy.Execute(() => Task.Factory.StartNew(() => obj.Do(), TaskCreationOptions.LongRunning));
}
Will this be called in parallel and retry each obj? So if myObjs[5].do() fails, will only that get retried while other objects just get executed once?
Also, am I supposed to use the ExecuteAsync() method which accepts Func<Task> instead of the Execute(Action) one as shown in the example? Do() is just a synchronous method, being launched in a separate thread.
Actual code looks like this where each() is just a foreach wrapper()
_consumers.ForEach(c => policy.Execute(() => Task.Factory.StartNew(() => c.Consume(startFromBeg), TaskCreationOptions.LongRunning)));
EDIT:
I tried the code:
class Foo
{
private int _i;
public Foo(int i)
{
_i = i;
}
public void Do()
{
//var rnd = new Random();
if (_i==2)
{
Console.WriteLine("err"+_i);
throw new Exception();
}
Console.WriteLine(_i);
}
}
var policy = Policy.Handle<Exception>().Retry(3);
var foos=Enumerable.Range(0, 5).Select(x => new Foo(x)).ToList();
foos.ForEach(c => policy.Execute(() => Task.Factory.StartNew(() => c.Do(), TaskCreationOptions.LongRunning)));
but am getting result:
0 1 err2 3 4 5
I thought it would retry 2 a few more times, but doesn't. Any idea why?
Whatever owns the tasks must wait for them somehow. Otherwise, exceptions will be ignored and the code will end before the tasks actually complete. So yes, you should probably be using policy.ExecuteAsync() instead. It would look something like this:
var tasks = myObjs
.Select(obj => Task.Factory.StartNew(() => obj.Do(), TaskCreationOptions.LongRunning))
.ToList();
// sometime later
await Task.WhenAll(tasks);
I'm having troubles testing a class that makes use of Observable.FromAsync<T>() and Observable.Switch<T>(). What it does is to wait for a trigger observable to produce a value, then it starts an async operation, and finally recollects all operations' results in a single output sequence. The gist of it is something like:
var outputStream = triggerStream
.Select(_ => Observable
.FromAsync(token => taskProducer.DoSomethingAsync(token)))
.Switch();
I put up some sanity check tests with the bare minimum parts to understand what's going on, here's the test with results in comments:
class test_with_rx : nspec
{
void Given_async_task_and_switch()
{
Subject<Unit> triggerStream = null;
TaskCompletionSource<long> taskDriver = null;
ITestableObserver<long> testObserver = null;
IDisposable subscription = null;
before = () =>
{
TestScheduler scheduler = new TestScheduler();
testObserver = scheduler.CreateObserver<long>();
triggerStream = new Subject<Unit>();
taskDriver = new TaskCompletionSource<long>();
// build stream under test
IObservable<long> streamUnderTest = triggerStream
.Select(_ => Observable
.FromAsync(token => taskDriver.Task))
.Switch();
/* Also tried with this Switch() overload
IObservable<long> streamUnderTest = triggerStream
.Select(_ => taskDriver.Task)
.Switch(); */
subscription = streamUnderTest.Subscribe(testObserver);
};
context["Before trigger"] = () =>
{
it["Should not notify"] = () => testObserver.Messages.Count.Should().Be(0);
// PASSED
};
context["After trigger"] = () =>
{
before = () => triggerStream.OnNext(Unit.Default);
context["When task completes"] = () =>
{
long result = -1;
before = () =>
{
taskDriver.SetResult(result);
//taskDriver.Task.Wait(); // tried with this too
};
it["Should notify once"] = () => testObserver.Messages.Count.Should().Be(1);
// FAILED: expected 1, actual 0
it["Should notify task result"] = () => testObserver.Messages[0].Value.Value.Should().Be(result);
// FAILED: of course, index out of bound
};
};
after = () =>
{
taskDriver.TrySetCanceled();
taskDriver.Task.Dispose();
subscription.Dispose();
};
}
}
In other tests I've done with mocks too, I can see that the Func passed to FromAsync is actually invoked (e.g. taskProducer.DoSomethingAsync(token)), but then it looks like nothing more follows, and the output stream doesn't produce the value.
I also tried inserting some Task.Delay(x).Wait(), or some taskDriver.Task.Wait() before hitting expectations, but with no luck.
I read this SO thread and I'm aware of schedulers, but at a first look I thought I didn't need them, no ObserveOn() is being used. Was I wrong? What am I missing? TA
Just for completeness, testing framework is NSpec, assertion library is FluentAssertions.
What you're hitting is a case of testing Rx and TPL together.
An exhaustive explanation can be found here but I'll try to give advice for your particular code.
Basically your code is working fine, but your test is not.
Observable.FromAsync will transform into a ContinueWith on the provided task, which will be executed on the taskpool, hence asynchronously.
Many ways to fix your test: (from ugly to complex)
Sleep after result set (note wait doesn't work because Wait doesn't wait for continuations)
taskDriver.SetResult(result);
Thread.Sleep(50);
Set the result before executing FromAsync (because FromAsync will return an immediate IObservable if the task is finished, aka will skip ContinueWith)
taskDriver.SetResult(result);
triggerStream.OnNext(Unit.Default);
Replace FromAsync by a testable alternative, e.g
public static IObservable<T> ToObservable<T>(Task<T> task, TaskScheduler scheduler)
{
if (task.IsCompleted)
{
return task.ToObservable();
}
else
{
AsyncSubject<T> asyncSubject = new AsyncSubject<T>();
task.ContinueWith(t => task.ToObservable().Subscribe(asyncSubject), scheduler);
return asyncSubject.AsObservable<T>();
}
}
(using either a synchronous TaskScheduler, or a testable one)
I'm trying to integrate some TPL async into a larger Rx chain using Observable.FromAsync, like in this small example:
using System;
using System.Reactive.Linq;
using System.Threading.Tasks;
namespace rxtest
{
class Program
{
static void Main(string[] args)
{
MainAsync().Wait();
}
static async Task MainAsync()
{
await Observable.Generate(new Random(), x => true,
x => x, x => x.Next(250, 500))
.SelectMany((x, idx) => Observable.FromAsync(async ct =>
{
Console.WriteLine("start: " + idx.ToString());
await Task.Delay(x, ct);
Console.WriteLine("finish: " + idx.ToString());
return idx;
}))
.Take(10)
.LastOrDefaultAsync();
}
}
}
However, I've noticed that this seems to start all the async tasks concurrently rather than doing them one at a time, which causes memory usage of the app to balloon. The SelectMany appears to be acting no different than a Merge.
Here, I see output like this:
start: 0
start: 1
start: 2
...
I'd like to see:
start: 0
finish: 0
start: 1
finish: 1
start: 2
finish: 2
...
How can I achieve this?
Change the SelectMany to a Select with a Concat:
static async Task MainAsync()
{
await Observable.Generate(new Random(), x => true,
x => x, x => x.Next(250, 500))
.Take(10)
.Select((x, idx) => Observable.FromAsync(async ct =>
{
Console.WriteLine("start: " + idx.ToString());
await Task.Delay(x, ct);
Console.WriteLine("finish: " + idx.ToString());
return idx;
}))
.Concat()
.LastOrDefaultAsync();
}
EDIT - I moved the Take(10) up the chain because the Generate won't block - so it stops this running away.
The Select projects each event into a stream representing an async task that will start on Subscription. Concat accepts a stream of streams and subscribes to each successive sub-stream when the previous has completed, concatenating all the streams into a single flat stream.