I'm working on some old code that does not have IAsyncEnumerable, so I was trying to mimic IAsyncEnumerable behavior with IObservable. The issue is that IObservable is push-based and is giving results in random order.
Here is a very simplified version of what I want to achieve
public static class Program
{
public static void Main()
{
Test().Wait();
Console.ReadLine();
}
private static async Task Test()
{
var l = await GetFirstObservable()
.SelectMany(i =>
{
return GetSecondObservable()
.Select(s => Tuple.Create(i, s));
})
.Take(3)
.ToList();
foreach (var item in l)
{
Console.WriteLine(item);
}
}
private static async Task<T> Echo<T>(T value)
{
Console.WriteLine(value);
await Task.Delay(1);
return value;
}
private static IObservable<int> GetFirstObservable()
{
return Observable.Create(async (IObserver<int> observer, CancellationToken cancellationToken) =>
{
foreach (var i in new [] {1, 2, 3})
{
if (cancellationToken.IsCancellationRequested)
{
return;
}
var e = await Echo(i);
if (cancellationToken.IsCancellationRequested)
{
return;
}
observer.OnNext(e);
}
});
}
private static IObservable<string> GetSecondObservable()
{
return Observable.Create(async (IObserver<string> observer, CancellationToken cancellationToken) =>
{
foreach (var i in new[] { "A", "B" })
{
if (cancellationToken.IsCancellationRequested)
{
return;
}
var e = await Echo(i);
if (cancellationToken.IsCancellationRequested)
{
return;
}
observer.OnNext(i);
}
});
}
}
I want the following results to be in console:
1
A
B
2
A
(1, A)
(1, B)
(2, A)
Instead, the first observable does not wait for the second observable to complete, and I get random results like:
1
A
2
A
3
B
A
B
(1, A)
(2, A)
(1, B)
Is it even possible to achieve this pull-like behavior, or should I just look for something else?
Related
I want to convert IEnumerable<Task<T>> to IObservable<T>. I found solution to this here:
IObservable<T> ToObservable<T>(IEnumerable<Task<T>> source)
{
return source.Select(t => t.ToObservable()).Merge();
}
It is perfectly ok for usual cases, but I need to handle exceptions, that could raise in that Tasks... So IObservable<T> should not be dead after first exception.
What I read, recommendation for this use case is to use some wrapper, that will carry actual value or error. So my attempt was
IObservable<Either<T, Exception>> ToObservable<T>(IEnumerable<Task<T>> source)
{
var subject = new Subject<Either<T, Exception>>();
foreach (var observable in GetIntsIEnumerable().Select(t => t.ToObservable()))
{
observable.Subscribe(i => subject.OnNext(i), e => subject.OnNext(e));
}
return subject;
}
With Either<T, Exception> borrowed from this article.
But this is not ok either, because OnCompleted() is not called. How should I solve it? I'm pretty new with Rx concept.
Here is full code for testing...
using System;
using System.Collections.Generic;
using System.Linq;
using System.Reactive.Linq;
using System.Reactive.Subjects;
using System.Reactive.Threading.Tasks;
using System.Threading;
using System.Threading.Tasks;
namespace Test
{
class Program
{
static Task Main()
{
SemaphoreSlim signal = new SemaphoreSlim(0, 1);
//GetInts1().Subscribe(
// i => Console.WriteLine($"OK: {i}"),
// e => Console.WriteLine($"ERROR: {e.Message}"),
// () => signal.Release());
GetInts2().Subscribe(r => Console.WriteLine(r.Match(
i => $"OK: {i}",
e => $"ERROR: {e.Message}")),
() => signal.Release());
return signal.WaitAsync();
}
static IObservable<int> GetInts1()
{
return GetIntsIEnumerable().Select(t => t.ToObservable()).Merge();
}
static IObservable<Either<int, Exception>> GetInts2()
{
var subject = new Subject<Either<int, Exception>>();
foreach (var observable in GetIntsIEnumerable().Select(t => t.ToObservable()))
{
observable.Subscribe(i => subject.OnNext(i), e => subject.OnNext(e));
}
return subject;
}
static IEnumerable<Task<int>> GetIntsIEnumerable()
{
Random rnd = new Random();
foreach (int i in Enumerable.Range(1, 10))
{
yield return Task.Run(async () =>
{
await Task.Delay(rnd.Next(0, 5000));
if (i == 6)
throw new ArgumentException();
return i;
});
}
}
}
/// <summary>
/// Functional data data to represent a discriminated
/// union of two possible types.
/// </summary>
/// <typeparam name="TL">Type of "Left" item.</typeparam>
/// <typeparam name="TR">Type of "Right" item.</typeparam>
public class Either<TL, TR>
{
private readonly TL left;
private readonly TR right;
private readonly bool isLeft;
public Either(TL left)
{
this.left = left;
this.isLeft = true;
}
public Either(TR right)
{
this.right = right;
this.isLeft = false;
}
public T Match<T>(Func<TL, T> leftFunc, Func<TR, T> rightFunc)
{
if (leftFunc == null)
{
throw new ArgumentNullException(nameof(leftFunc));
}
if (rightFunc == null)
{
throw new ArgumentNullException(nameof(rightFunc));
}
return this.isLeft ? leftFunc(this.left) : rightFunc(this.right);
}
/// <summary>
/// If right value is assigned, execute an action on it.
/// </summary>
/// <param name="rightAction">Action to execute.</param>
public void DoRight(Action<TR> rightAction)
{
if (rightAction == null)
{
throw new ArgumentNullException(nameof(rightAction));
}
if (!this.isLeft)
{
rightAction(this.right);
}
}
public TL LeftOrDefault() => this.Match(l => l, r => default);
public TR RightOrDefault() => this.Match(l => default, r => r);
public static implicit operator Either<TL, TR>(TL left) => new Either<TL, TR>(left);
public static implicit operator Either<TL, TR>(TR right) => new Either<TL, TR>(right);
}
}
There's a built-in mechanism for handling errors like this. Simply use the .Materialize() operator which changes an IObservable<T> to an IObservable<Notification<T>> and allows errors and completions to be viewed as normal values.
So, as an example, Observable.Return<int>(42) produces a value 42 and a completion, but Observable.Return<int>(42).Materialize() produces a value Notification.CreateOnNext<int>(42), followed by a value Notification.CreateOnCompleted<int>(), followed by a normal completion.
If you have a sequence that produces an error then you effectively get a value Notification.CreateOnError<T>(exception) followed by a normal completion.
This all means that you can change your code like this:
IObservable<Notification<T>> ToObservable<T>(IEnumerable<Task<T>> source)
{
return source.Select(t => t.ToObservable().Materialize()).Merge();
}
Your test code is a little bit complicated for my liking. You should never need to use a SemaphoreSlim nor a Subject in the way that you're using them.
I've written my own test code.
void Main()
{
var r = new Random();
IEnumerable<Task<int>> source =
Enumerable
.Range(0, 10).Select(x => Task.Factory.StartNew(() =>
{
Thread.Sleep(r.Next(10000));
if (x % 3 == 0) throw new NotSupportedException($"Failed on {x}");
return x;
}));
IObservable<Notification<int>> query = source.ToObservable();
query
.Do(x =>
{
if (x.Kind == NotificationKind.OnError)
{
Console.WriteLine(x.Exception.Message);
}
})
.Where(x => x.Kind == NotificationKind.OnNext) // Only care about vales
.Select(x => x.Value)
.Subscribe(x => Console.WriteLine(x), () => Console.WriteLine("Done."));
}
public static class Ex
{
public static IObservable<Notification<T>> ToObservable<T>(this IEnumerable<Task<T>> source)
{
return source.Select(t => t.ToObservable().Materialize()).Merge();
}
}
A typical run of that code produces:
Failed on 3
2
5
4
Failed on 0
Failed on 9
Failed on 6
7
1
8
Done.
The Rx library contains a Merge overload that merges tasks directly and efficiently, instead of converting each task to an intermediate throw-away observable sequence:
// Merges results from all source tasks into a single observable sequence.
public static IObservable<TSource> Merge<TSource>(
this IObservable<Task<TSource>> sources);
You could use this operator for the implementation of the ToObservable method like this:
IObservable<Either<T, Exception>> ToObservable<T>(IEnumerable<Task<T>> source)
{
return source
.Select(async task =>
{
try { return new Either<T, Exception>(await task); }
catch (Exception ex) { return new Either<T, Exception>(ex); }
})
.ToObservable()
.Merge();
}
You can place the ToObservable operator before or after the Select operator, it doesn't make any difference.
Btw there is available a minimalistic library (Try by Stephen Cleary) that contains a Try<T> type, which is similar in functionality to the Either type, but is specialized for having an Exception as the second type (as an Either<T, Exception>). Using this library, you could implement the ToObservable method like this:
IObservable<Try<T>> ToObservable<T>(IEnumerable<Task<T>> source)
{
return source
.Select(task => Try.Create(() => task))
.ToObservable()
.Merge();
}
Here is the definition of the Try.Create method:
// Executes the specified function, and wraps either the result or the exception.
public static Task<Try<T>> Create<T>(Func<Task<T>> func);
I need to execute many methods at the same time and the result of all, concatenate them in a single list. In the following example, I wait 3 seconds for each method, but in one of them I added a sleep (10 seconds) to check the result and it is not the expected one. The method never cancels and waits for those 10 seconds. What is the problem? Thank you!
var result = await Task.Run(() => Load<MyCustomClass>(OneMethod(), OtherMethod()));
private List<T> OneMethod()
{
return new List<T>();
}
private List<T> OtherMethod()
{
Thread.Sleep(10000);
return new List<T>();
}
private async Task<List<T>> Load<T>(params List<T>[] taskList)
{
try
{
return (await Task.WhenAll(taskList.Select(x =>
Task.Run(() => x, new CancellationTokenSource(3000).Token)))).SelectMany(x => x).ToList();
}
catch (Exception currentException)
{
//BLA BLA BLA
}
return new List<T>();
}
You must pass the CancellationToken to the methods and check if cancellationToken is requested or directly raise an exception.
var t = new CancellationTokenSource(3000).Token
var result = await Task.Run(() => Load<MyCustomClass>(OneMethod(t), OtherMethod(t)));
private List<T> OneMethod(CancellationToken t)
{
token.ThrowIfCancellationRequested();
return new List<T>();
}
private List<T> OtherMethod(CancellationToken t)
{
Thread.Sleep(10000);
token.ThrowIfCancellationRequested();
// or you can check if cancellation is requested
// if (token.IsCancellationRequested) ...
return new List<T>();
}
private async Task<List<T>> Load<T>(params List<T>[] taskList)
{
try
{
return (await Task.WhenAll(taskList.Select(x =>
Task.Run(() => x)))).SelectMany(x => x).ToList();
}
catch (Exception currentException)
{
//BLA BLA BLA
}
return new List<T>();
}
See this question
I have a multi-threaded application that has to perform 3 different categories of work items. Category A is the highest priority items, Category B items comes after A and Category C items comes after B. These work items are queued to thread pool using Tasks. Let's say, there are 10 category C items already in queue and then a category B item is added. In this case, I would like category B item to be processed before any of the category C item. Is there anyway of accomplishing this?
You can implement it by creating your own queue process. This is just a code mockup.
Create an object like this
public class PrioritizableTask
{
public PrioritizableTask(Task task, int taskPriority)
{
Task = task;
Priority = taskPriority;
}
public int Priority { get; private set; }
public Task Task { get; private set; }
}
And then create another collection class and implement a new method on it, something like this.
public class PrioritizableTasksCollection : IList<PrioritizableTask>
{
private static readonly List<PrioritizableTask> runners = new List<PrioritizableTask>();
public void Add(PrioritizableTask item)
{
runners.Add(item);
}
public Task GetNextPriority()
{
var priorityTask = runners.OrderBy(x => x.Priority).FirstOrDefault();
return priorityTask != null ? priorityTask.Task : null;
}
}
Consume like
PrioritizableTasksCollection executors = new PrioritizableTasksCollection();
executors.Add(new PrioritizableTask(new Task(() => { }), 4));
executors.Add(new PrioritizableTask(new Task(() => { }), 3));
executors.Add(new PrioritizableTask(new Task(() => { }), 7));
executors.Add(new PrioritizableTask(new Task(() => { }), 5));
executors.Add(new PrioritizableTask(new Task(() => { }), 1));
executors.Add(new PrioritizableTask(new Task(() => { }), 2));
Task executeNext = executors.GetNextPriority();
Implement your own deleting on the collection.
I've been looking at your problem and i did not find a built-in thread-safe sorted collection.
So i built a basic thread-safe SortedSet<int> wrapper class.
Sorted Set
public class MyThreadSafeSortedSet
{
private SortedSet<int> _set = new SortedSet<int>(new MyComparer());
private readonly object _locker = new object();
public void Add(int value)
{
lock (_locker)
{
_set.Add(value);
}
}
public int? Take()
{
lock (_locker)
{
if (_set.Count == 0)
return null;
var item = _set.First();
_set.Remove(item);
return item;
}
}
}
I built a custom comparer which prefers even numbers
public class MyComparer : Comparer<int>
{
public override int Compare(int x, int y)
{
if (x % 2 == 0)
{
if (y % 2 == 0)
return x - y;
else
return -1;
}
else
{
if (y % 2 == 0)
return 1;
else
return x - y;
}
}
}
And finally two threads. One to produce items; the other one to take them
static void Main(string[] args)
{
MyThreadSafeSortedSet queue = new MyThreadSafeSortedSet();
var task1 = Task.Run(() =>
{
Random r = new Random();
for (int i = 0; i < 15; i++)
{
Task.Delay(100).Wait();
var randomNumber = r.Next();
queue.Add(randomNumber);
}
Console.WriteLine("I'm done adding");
});
var task2 = Task.Run(() =>
{
Random r = new Random();
while (true)
{
var delay = r.Next(500);
Task.Delay(delay).Wait();
var item = queue.Take();
Console.WriteLine("Took: {0}", item);
if (item == null)
break;
}
});
Task.WaitAll(task2);
}
You can change the specialized SortedSet and custom comparer for your own classes.
Hope it helped
Please look my version of solution based on BinarySearch method of List class.
enum CategoryOfWorkItem: int { C = 0, B, A };
struct WorkItem : IComparer<WorkItem>
{
public CategoryOfWorkItem Category;
public int Compare(WorkItem x, WorkItem y)
{
return x.Category - y.Category;
}
public void AddTo(List<WorkItem> list)
{
int i = list.BinarySearch(this, this);
if (i < 0) i = ~i;
list.Insert(i, this);
}
}
Example of usage
List<WorkItem> list = new List<WorkItem>();
Task.Run(() =>
{
Random rand = new Random();
for (int i = 0; i < 20; i++)
{
WorkItem item = new WorkItem();
switch (rand.Next(0, 3))
{
case 0: item.Category = CategoryOfWorkItem.A; break;
case 1: item.Category = CategoryOfWorkItem.B; break;
case 2: item.Category = CategoryOfWorkItem.C; break;
}
lock (list)
{
item.AddTo(list);
}
Task.Delay(rand.Next(100, 1000)).Wait();
Console.WriteLine("Put {0}", item.Category);
}
Console.WriteLine("Putting finished.");
});
Task.WaitAll(Task.Run(() =>
{
Random rand = new Random();
while (true)
{
WorkItem item;
Task.Delay(rand.Next(500, 1000)).Wait();
lock (list)
{
if (list.Count < 1) break;
item = list[list.Count - 1];
list.RemoveAt(list.Count - 1);
}
Console.WriteLine("Get {0}", item.Category);
}
Console.WriteLine("Getting finished.");
}));
I have an IObservable<string> and several observers that handle strings based on some condition:
observable.Subscribe(s => { if (s.StartsWith("a")) {...} });
observable.Subscribe(s => { if (s.StartsWith("b")) {...} });
observable.Subscribe(s => { if (s.StartsWith("c")) {...} });
observable.Subscribe(s => { if (s.StartsWith("d")) {...} });
....
This is a simplified example (the condition is more complex and the observed events aren't strings) but you get the idea.
I'd like to have an IObserver<string> that catches all strings that are not handled by any other observer. Observers with different conditions (i.e.: StartsWith("e")) can be added at any time and the set of conditions does not overlap.
Is this scenario somehow supported? Or do I have to mark observed strings as handled and subscribe to unhandled strings once all other observers have tried (and how do I implement that)?
I've got two approaches.
The first provides a way to chain together the predicate/action pairs to "syphon" off values that match. It follows the Rx operator style.
I can write this:
observable
.Syphon(s => s.StartsWith("a"), s => { })
.Syphon(s => s.StartsWith("b"), s => { })
.Syphon(s => s.StartsWith("c"), s => { })
.Syphon(s => s.StartsWith("d"), s => { })
.Subscribe(s => { /* otherwise */ });
If I have this extension method:
public static IObservable<T> Syphon<T>(
this IObservable<T> source,
Func<T, bool> predicate,
Action<T> action)
{
if (source == null) throw new ArgumentNullException("source");
if (predicate == null) throw new ArgumentNullException("predicate");
if (action == null) throw new ArgumentNullException("action");
return Observable.Create<T>(o =>
source.Subscribe(
t =>
{
if (predicate(t))
{
action(t);
}
else
{
o.OnNext(t);
}
},
ex =>
o.OnError(ex),
() =>
o.OnCompleted()));
}
It doesn't allow you to add and remove predicate/action pairs on the fly, but it is a fairly simple operator that might be useful.
To have the full add/remove functionality I have come up with this approach:
Func<Func<string, bool>, Action<string>, IDisposable> add;
observable
.Syphon(out add)
.Subscribe(s => { /* otherwise */ });
var startsWithA = add(s => s.StartsWith("a"), s => { /* a */ });
var startsWithB = add(s => s.StartsWith("b"), s => { /* b */ });
startsWithA.Dispose();
var startsWithC = add(s => s.StartsWith("c"), s => { /* c */ });
var startsWithD = add(s => s.StartsWith("d"), s => { /* d */ });
startsWithC.Dispose();
startsWithB.Dispose();
startsWithD.Dispose();
The .Syphon(out add) extension method overload allows the method to effectively return two results - the normal return value is the IObservable<T> and the second comes out as a Func<Func<T, bool>, Action<T>, IDisposable>. This second return value allows new predicate/action pairs to be added to the syphon operator and then removed by calling Dispose on the returned subscription - very Rx-ish.
Here's the extension method:
public static IObservable<T> Syphon<T>(
this IObservable<T> source,
out Func<Func<T, bool>, Action<T>, IDisposable> subscriber)
{
if (source == null) throw new ArgumentNullException("source");
var pas = new List<Tuple<Func<T, bool>, Action<T>>>();
subscriber = (p, a) =>
{
lock (pas)
{
var tuple = Tuple.Create(p, a);
pas.Add(tuple);
return Disposable.Create(() =>
{
lock (pas)
{
pas.Remove(tuple);
}
});
}
};
return Observable.Create<T>(o =>
source.Subscribe(
t =>
{
Action<T> a = null;
lock (pas)
{
var pa = pas.FirstOrDefault(x => x.Item1(t));
if (pa != null)
{
a = pa.Item2;
}
}
if (a != null)
{
a(t);
}
else
{
o.OnNext(t);
}
},
ex =>
o.OnError(ex),
() =>
o.OnCompleted()));
}
I tested the code with this:
var xs = Observable.Interval(TimeSpan.FromSeconds(0.2));
Func<Func<long, bool>, Action<long>, IDisposable> subscriber;
xs
.Syphon(out subscriber)
.Subscribe(x => Console.WriteLine(x));
var divBy3 = subscriber(
x => x % 3 == 0,
x => Console.WriteLine("divBy3"));
Thread.Sleep(2000);
var divBy2 = subscriber(
x => x % 2 == 0,
x => Console.WriteLine("divBy2"));
Thread.Sleep(2000);
divBy3.Dispose();
Thread.Sleep(2000);
divBy2.Dispose();
Thread.Sleep(10000);
And it produced:
divBy3
1
2
divBy3
4
5
divBy3
7
8
divBy3
divBy2
11
divBy3
13
divBy2
divBy3
divBy2
17
divBy3
19
divBy2
21
divBy2
23
divBy2
25
divBy2
27
divBy2
29
30
31
32
...
And that seemed right. Let me know if this solves it for you.
One option is to make your subscribers to be observable as well. So what these subscribers does is if that they don't handle the value then they emit it through their observable interface and then the last subscriber (that handle all not used values) will be a single ton object that subscribes to each of the observable interface of the other subscribers. Something like:
public class MyObserver : IObserver<string>, IObservable<string>
{
Subject<string> s = new Subject<string>();
public MyObserver(IObserver<string> obs)
{
s.Subscribe(obs);
}
public void OnCompleted()
{ }
public void OnError(Exception error)
{ }
public void OnNext(string value)
{
//If condition matches then else dont do on next
s.OnNext(value);
}
public IDisposable Subscribe(IObserver<string> observer)
{
return s.Subscribe(observer);
}
}
public class LastObserver : IObserver<string>
{
public void OnCompleted()
{ }
public void OnError(Exception error)
{ }
public void OnNext(string value)
{ //Do something with not catched value
}
}
static LastObserver obs = new LastObserver();
static void Main()
{
var timer = Observable.Interval(TimeSpan.FromSeconds(1)).Select(i => i.ToString());
timer.Subscribe(new MyObserver(obs));
timer.Subscribe(new MyObserver(obs));
timer.Subscribe(new MyObserver(obs));
}
I don't know of any out of the box way to do this but I would do it as under
class ConditionAction
{
public Predicate<string> Condition {get; set; }
public Action<string> Action {get; set; }
}
var conditions = new ConditionAction[]{action1, action2, action3};
foreach (var condition in conditions)
observable.Where(condition.Condition).Subscribe(condition.Action);
.....
observable.Where(s=>!conditions.Any(c=>c.Condition(s))).Subscribe(...);
The following program
static IEnumerable<Action> Create()
{
foreach (var i in Enumerable.Range(0, 2))
{
yield return () => { Console.Write(i); };
}
}
static void Main(string[] args)
{
foreach (var func in Create())
{
func();
}
Console.ReadLine();
}
ouputs
01
And this program
static event Action SomethingHappened;
static void Register()
{
foreach (var i in Enumerable.Range(0, 2))
{
SomethingHappened += () => { Console.Write(i); };
}
}
static void Main(string[] args)
{
Register();
SomethingHappened();
Console.ReadLine();
}
outputs
11
Why is that so? How to make the program 2 output 01?
You're capturing the loop variable in your lambda expression. That means when the delegate is finally invoked, it will use the latest value of the variable... which will always be 1. Try this:
foreach (var i in Enumerable.Range(0, 2))
{
int copy = i;
SomethingHappened += () => { Console.Write(copy); };
}
... then read Eric Lippert's blog post about it.
In the second program the variable i is captured by the lambda. To get the correct behavior make a local copy before using it in the lambda. E.g.
foreach (var i in Enumerable.Range(0, 2))
{
var local = i;
SomethingHappened += () => { Console.Write(local); };
}