Develop Reference

DistinctUntilChanged fires multiple times on multiple subscribers

I have one observable (mainSequence). If a condition is meet it should invoke an async method once until the condition changes. The methods return value will indicate success.
On failure I have a subscription which will inform the user.
Other observable are likely to subscribe to the mainSequence and have a similar error handling pattern.
But the consecutive observers to mainSequence will cause to invoke the mainSequence again. I only would like to have it invoked once hence my DistinctUntilChanged.
The example below outputs:
Working on 6
Working on 6
Working on 100
Working on 6
Working on 101
The output I want is:
Working on 6
Working on 100
Working on 101
I'm missing an reactive operator on my mainSequence, I just don't know which one.
public static void Main()
{
bool IsNumberOk(int n) => n > 5;
Task<bool> DoSomethingAsync(int n)
{
Console.WriteLine($"Working on {n}");
return Task.FromResult(true);
}
var mainSequence = Observable.Range(0, 10)
.Where(IsNumberOk)
.DistinctUntilChanged(IsNumberOk)
.SelectMany(DoSomethingAsync);
// sequence one error handling
mainSequence.Where(x => !x).Subscribe(_ => Console.WriteLine($"Something went wrong with {nameof(mainSequence)}"));
for (var i = 0; i < 2; i++)
{
var iTemp = 100 + i;
var consecutive = mainSequence
.Where(x => x) // if no error on mainSequence
.Select(_ => iTemp)
.DistinctUntilChanged()
.SelectMany(DoSomethingAsync);
consecutive.Where(x => !x).Subscribe(_ => Console.WriteLine($"Something went wrong with {iTemp}"));
}
}

You have misunderstanding with regard to the distinction between an observable and a subscription. They are two distinct things.
The best parallel, in my mind, is that an observable is like a class and a subscription is like an instance of a class. Like a class, the observable is defined once. Each subscription is a new instance of the observable.
Let's take this code - somewhat cut-down from your code in the question.
Task<int> DoSomethingAsync(int n)
{
Console.WriteLine($"Working on {n}");
return Task.FromResult(-n);
}
IObservable<int> mainSequence =
Observable
.Range(0, 3)
.SelectMany(DoSomethingAsync);
That's a single observable.
Now let's do this:
IDisposable mainSubscription1 =
mainSequence
.Subscribe(x => Console.WriteLine($"(1){nameof(mainSequence)}OnNext({x})"));
IDisposable mainSubscription2 =
mainSequence
.Subscribe(x => Console.WriteLine($"(2){nameof(mainSequence)}OnNext({x})"));
I have created two subscriptions, so I get two completely distinct instances of the observable. They run entirely separate of each other. In fact, Observable.Range outputs its values immediately, so each subscription blocks until it is complete. You get this output:
Working on 0
(1)mainSequenceOnNext(0)
Working on 1
(1)mainSequenceOnNext(-1)
Working on 2
(1)mainSequenceOnNext(-2)
Working on 0
(2)mainSequenceOnNext(0)
Working on 1
(2)mainSequenceOnNext(-1)
Working on 2
(2)mainSequenceOnNext(-2)
You can get Observable.Range to not block like this:
IObservable<int> mainSequence =
Observable
.Range(0, 3, Scheduler.Default)
.SelectMany(DoSomethingAsync);
But you still have two completely independent instances of the observable running. You get something like this:
Working on 0
Working on 0
(1)mainSequenceOnNext(0)
Working on 1
(2)mainSequenceOnNext(0)
Working on 1
(1)mainSequenceOnNext(-1)
Working on 2
(2)mainSequenceOnNext(-1)
Working on 2
(1)mainSequenceOnNext(-2)
(2)mainSequenceOnNext(-2)
Now, if you want to share a single observable then you need to Publish it and Connect to the published observable to get the values flowing.
Here's the full code:
IConnectableObservable<int> mainSequence =
Observable
.Range(0, 3, Scheduler.Default)
.SelectMany(DoSomethingAsync)
.Publish();
IDisposable mainSubscription1 =
mainSequence
.Subscribe(x => Console.WriteLine($"(1){nameof(mainSequence)}OnNext({x})"));
IDisposable mainSubscription2 =
mainSequence
.Subscribe(x => Console.WriteLine($"(2){nameof(mainSequence)}OnNext({x})"));
IDisposable mainConnection =
mainSequence
.Connect();
Now when I run that, the two subscriptions don't start producing values until the .Connect() is called.
You get this:
Working on 0
(1)mainSequenceOnNext(0)
(2)mainSequenceOnNext(0)
Working on 1
(1)mainSequenceOnNext(-1)
(2)mainSequenceOnNext(-1)
Working on 2
(1)mainSequenceOnNext(-2)
(2)mainSequenceOnNext(-2)
Now if I had to get your code working, here's what it would look like:
public static void Main()
{
bool IsNumberOk(int n) => n > 5;
Task<bool> DoSomethingAsync(int n)
{
Console.WriteLine($"Working on {n}");
return Task.FromResult(true);
}
var mainSequence =
Observable
.Range(0, 10)
.Where(IsNumberOk)
.DistinctUntilChanged(IsNumberOk)
.SelectMany(DoSomethingAsync)
.Publish();
mainSequence
.Where(x => !x)
.Subscribe(_ => Console.WriteLine($"Something went wrong with {nameof(mainSequence)}"));
for (var i = 0; i < 2; i++)
{
var iTemp = 100 + i;
var consecutive =
mainSequence
.Where(x => x)
.Select(_ => iTemp)
.DistinctUntilChanged()
.SelectMany(DoSomethingAsync);
consecutive
.Where(x => !x)
.Subscribe(_ => Console.WriteLine($"Something went wrong with {iTemp}"));
}
IDisposable mainConnection =
mainSequence
.Connect();
}
It now produces this:
Working on 6
Working on 100
Working on 101

rx.net locking up from use of ToEnumerable

I am trying to convert the below statement so that I can get the key alongside the selected list:
var feed = new Subject<TradeExecuted>();
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(TimeSpan.FromSeconds(5)))
.SelectMany(x => x.ToList())
.Select(trades => Observable.FromAsync(() => Mediator.Publish(trades, cts.Token)))
.Concat() // Ensure that the results are serialized.
.Subscribe(cts.Token); // Check status of calls.
The above works, whereas the below does not - when I try and itterate over the list, it locks up.
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(timespan))
.Select(x => Observable.FromAsync(() =>
{
var list = x.ToEnumerable(); // <---- LOCK UP if we use list.First() etc
var aggregate = AggregateTrades(x.Key.Symbol, x.Key.AccountId, x.Key.Tenant, list);
return Mediator.Publish(aggregate, cts.Token);
}))
.Concat()
.Subscribe(cts.Token); // Check status of calls.
I am clearly doing something wrong and probably horrific!
Going back to the original code, how can I get the Key alongside the enumerable list (and avoiding the hack below)?
As a sidenote, the below code works but it a nasty hack where I get the keys from the first list item:
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(TimeSpan.FromSeconds(5)))
.SelectMany(x => x.ToList())
.Select(trades => Observable.FromAsync(() =>
{
var firstTrade = trades.First();
var aggregate = AggregateTrades(firstTrade.Execution.Contract.Symbol, firstTrade.Execution.AccountId, firstTrade.Tenant, trades);
return Mediator.Publish(aggregate, cts.Token);
}))
.Concat() // Ensure that the results are serialized.
.Subscribe(cts.Token); // Check status of calls.

All versions of your code suffer from trying to eagerly evaluate the grouped sub-observable. Since in v1 and v3 your group observable will run a maximum of 5 seconds, that isn't horrible/awful, but it's still not great. In v2, I don't know what timespan is, but assuming it's 5 seconds, you have the same problem: Trying to turn the grouped sub-observable into a list or an enumerable means waiting for the sub-observable to complete, blocking the thread (or the task).
You can fix this by using the Buffer operator to lazily evaluate the grouped sub-observable:
var timespan = TimeSpan.FromSeconds(5);
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(timespan))
.SelectMany(x => x
.Buffer(timespan)
.Select(list => Observable.FromAsync(() =>
{
var aggregate = AggregateTrades(x.Key.Symbol, x.Key.AccountId, x.Key.Tenant, list));
return Mediator.Publish(aggregate, cts.Token);
}))
)
.Concat() // Ensure that the results are serialized.
.Subscribe(cts.Token); // Check status of calls.
This essentially means that until timespan is up, the items in the group by gather in a list inside Buffer. Once timespan is up, they're released as a list, and the mediator publish happens.

Rx.Net - process groups asynchronously and in parallel with a constrained concurrency

Playing with System.Reactive trying to resolve the next task -
Break an incoming stream of strings into groups
Items in each group must be processed asynchronously and sequentially
Groups must be processed in parallel
No more than N groups must be processed at the same time
Ideally, w/o using sync primitives
Here is the best I've figured out so far -
TaskFactory taskFactory = new (new LimitedConcurrencyLevelTaskScheduler(2));
TaskPoolScheduler scheduler = new (taskFactory);
source
.GroupBy(item => item)
.SelectMany(g => g.Select(item => Observable.FromAsync(() => onNextAsync(item))).ObserveOn(scheduler).Concat())
.Subscribe();
Any idea how to achieve it w/o a scheduler? Couldn't make it work via Merge()

The easiest way to enforce the "No more than N groups must be processed at the same time" limitation, is probably to use a SemaphoreSlim. So instead of this:
.SelectMany(g => g.Select(item => Observable.FromAsync(() => onNextAsync(item))).Concat())
...you can do this:
var semaphore = new SemaphoreSlim(N, N);
//...
.SelectMany(g => g.Select(item => Observable.FromAsync(async () =>
{
await semaphore.WaitAsync();
try { return await onNextAsync(item); }
finally { semaphore.Release(); }
})).Merge(1))
Btw in the current Rx version (5.0.0) I don't trust the Concat operator, and I prefer to use the Merge(1) instead.

To solve this problem using exclusively Rx tools, ideally you would like to have something like this:
source
.GroupBy(item => item.Key)
.Select(group => group.Select(
item => Observable.FromAsync(() => ProcessAsync(item))).Merge(1))
.Merge(maxConcurrent: N)
.Wait();
The inner Merge(1) would enforce the exclusive processing within each group, and the outer Merge(N) would enforce the global maximum concurrency policy. Unfortunately this doesn't work because the outer Merge(N) restricts the subscriptions to the inner sequences (the IGroupedObservable<T>s), not to their individual elements. This is not what you want. The result will be that only the first N groups to be processed, and the elements of all other groups will be ignored. The GroupBy operator creates hot subsequences, and if you don't subscribe to them immediately you'll lose elements.
In order for the outer Merge(N) to work as desired, you'll have to merge freely all the inner sequences that are produced by the Observable.FromAsync, and have some other mechanism to serialize the processing of each group. One idea is to implement a special Select operator that emits an Observable.FromAsync only after the previous one is completed. Below is such an implementation, based on the Zip operator. The Zip operator maintains internally two hidden buffers, so that it can produce pairs from two sequences that might emit elements with different frequences. This buffering is exactly what we need in order to avoid losing elements.
private static IObservable<IObservable<TResult>> SelectOneByOne<TSource, TResult>(
this IObservable<TSource> source,
Func<TSource, IObservable<TResult>> selector)
{
var subject = new BehaviorSubject<Unit>(default);
var synchronizedSubject = Observer.Synchronize(subject);
return source
.Zip(subject, (item, _) => item)
.Select(item => selector(item).Do(
_ => { },
_ => synchronizedSubject.OnNext(default),
() => synchronizedSubject.OnNext(default)));
}
The BehaviorSubject<T> contains initially one element, so the first pair will be produced immediately. The second pair will not be produced before the first element has been processed. The same with the third pair and second element, etc.
You could then use this operator to solve the problem like this:
source
.GroupBy(item => item.Key)
.SelectMany(group => group.SelectOneByOne(
item => Observable.FromAsync(() => ProcessAsync(item))))
.Merge(maxConcurrent: N)
.Wait();
The above solution is presented only for the purpose of answering the question. I don't think that I would trust it in a production environment.

Observe values not seen in other observers

I have an observable that emits unique values e.g.
var source=Observable.Range(1,100).Publish();
source.Connect();
I want to observe its values from e.g. two observers but each observer to get notified only for values not seen in other observers.
So if first observer contains the value 10 the second observer should never get notified for the 10 value.
Update
I chose #Asti`s answer cause it was first and although buggy it pointed to the right direction and up-voted #Shlomo's answer. Too bad I cannot accept both answers as #Shlomo answer was more correct and I really appreciate all his help we get on this tag.

Observables aren't supposed to behave differently for different observers; a better approach would be to give each observer its own filtered observable.
That being said, if your constraints require that you need this behavior in a single observable - we can use a Round-Robin method.
public static IEnumerable<T> Repeat<T>(this IEnumerable<T> source)
{
for (; ; )
foreach (var item in source.ToArray())
yield return item;
}
public static IObservable<T> RoundRobin<T>(this IObservable<T> source)
{
var subscribers = new List<IObserver<T>>();
var shared = source
.Zip(subscribers.Repeat(), (value, observer) => (value, observer))
.Publish()
.RefCount();
return Observable.Create<T>(observer =>
{
subscribers.Add(observer);
var subscription =
shared
.Where(pair => pair.observer == observer)
.Select(pair => pair.value)
.Subscribe(observer);
var dispose = Disposable.Create(() => subscribers.Remove(observer));
return new CompositeDisposable(subscription, dispose);
});
}
Usage:
var source = Observable.Range(1, 100).Publish();
var dist = source.RoundRobin();
dist.Subscribe(i => Console.WriteLine($"One sees {i}"));
dist.Subscribe(i => Console.WriteLine($"Two sees {i}"));
source.Connect();
Result:
One sees 1
Two sees 2
One sees 3
Two sees 4
One sees 5
Two sees 6
One sees 7
Two sees 8
One sees 9
Two sees 10
If you already have a list of observers, the code becomes much simpler.

EDIT: #Asti fixed his bug, and I fixed mine based on his answer. Our answers are now largely similar. I have an idea how to do a purely reactive one, if I have time I'll post that later.
Fixed code:
public static IObservable<T> RoundRobin2<T>(this IObservable<T> source)
{
var subscribers = new BehaviorSubject<ImmutableList<IObserver<T>>>(ImmutableList<IObserver<T>>.Empty);
ImmutableList<IObserver<T>> latest = ImmutableList<IObserver<T>>.Empty;
subscribers.Subscribe(l => latest = l);
var shared = source
.Select((v, i) => (v, i))
.WithLatestFrom(subscribers, (t, s) => (t.v, t.i, s))
.Publish()
.RefCount();
return Observable.Create<T>(observer =>
{
subscribers.OnNext(latest.Add(observer));
var dispose = Disposable.Create(() => subscribers.OnNext(latest.Remove(observer)));
var sub = shared
.Where(t => t.i % t.s.Count == t.s.FindIndex(o => o == observer))
.Select(t => t.v)
.Subscribe(observer);
return new CompositeDisposable(dispose, sub);
});
}
Original answer:
I upvoted #Asti's answer, because he's largely correct: Just because you can, doesn't mean you should. And his answer largely works, but it's subject to a bug:
This works fine:
var source = Observable.Range(1, 20).Publish();
var dist = source.RoundRobin();
dist.Subscribe(i => Console.WriteLine($"One sees {i}"));
dist.Take(1).Subscribe(i => Console.WriteLine($"Two sees {i}"));
This doesn't:
var source = Observable.Range(1, 20).Publish();
var dist = source.RoundRobin();
dist.Take(1).Subscribe(i => Console.WriteLine($"One sees {i}"));
dist.Subscribe(i => Console.WriteLine($"Two sees {i}"));
Output is:
One sees 1
Two sees 1
Two sees 2
Two sees 3
Two sees 4
...
I first thought the bug is Halloween related, but now I'm not sure. The .ToArray() in Repeat should take care of that. I also wrote a pure-ish observable implementation which has the same bug. This implementation doesn't guarantee a perfect Round Robin, but that wasn't in the question:
public static IObservable<T> RoundRobin2<T>(this IObservable<T> source)
{
var subscribers = new BehaviorSubject<ImmutableList<IObserver<T>>>(ImmutableList<IObserver<T>>.Empty);
ImmutableList<IObserver<T>> latest = ImmutableList<IObserver<T>>.Empty;
subscribers.Subscribe(l => latest = l);
var shared = source
.Select((v, i) => (v, i))
.WithLatestFrom(subscribers, (t, s) => (t.v, t.i, s))
.Publish()
.RefCount();
return Observable.Create<T>(observer =>
{
subscribers.OnNext(latest.Add(observer));
var dispose = Disposable.Create(() => subscribers.OnNext(latest.Remove(observer)));
var sub = shared
.Where(t => t.i % t.s.Count == t.s.FindIndex(o => o == observer))
.Select(t => t.v)
.Subscribe(observer);
return new CompositeDisposable(dispose, sub);
});
}

This is a simple distributed queue implementation using TPL Dataflow. But with respect to different observers not seeing the same value, there's little chance of it behaving incorrectly. It's not round-robin, but actually has back-pressure semantics.
public static IObservable<T> Distribute<T>(this IObservable<T> source)
{
var buffer = new BufferBlock<T>();
source.Subscribe(buffer.AsObserver());
return Observable.Create<T>(observer =>
buffer.LinkTo(new ActionBlock<T>(observer.OnNext, new ExecutionDataflowBlockOptions { BoundedCapacity = 1 })
);
}
Output
One sees 1
Two sees 2
One sees 3
Two sees 4
One sees 5
One sees 6
One sees 7
One sees 8
One sees 9
One sees 10
I might prefer skipping Rx entirely and just using TPL Dataflow.

Adding an observable sequence after subscription

We are using Rx to monitor activity within our silverlight application so that we can display a message to the user after a period of inactivity.
We are turning events (mouse moves etc.) into observables and then merging the observables together to create a single (allActivity) observable. We then throttle the allActivity observable using a timespan and something subscribes to be notified when the system has been inactive for a period of time.
How can I add a new observable/ sequence to this after the subscription (so that the subscription picks this up without unsubscribing and resubscribing).
e.g. merge several sequences together, throttle, subscribe. Now add an additional sequence to the observable that has been subscribed to.
Example code:
private IObservable<DateTime> allActivity;
public void CreateActivityObservables(UIElement uiElement)
{
// Create IObservables of event types we are interested in and project them as DateTimes
// These are our observables sequences that can push data to subscribers/ observers
// NB: These are like IQueryables in the sense that they do not iterate over the sequence just provide an IObservable type
var mouseMoveActivity = Observable.FromEventPattern<MouseEventHandler, MouseEventArgs>(h => uiElement.MouseMove += h, h => uiElement.MouseMove -= h)
.Select(o => DateTime.Now);
var mouseLeftButtonActivity = Observable.FromEventPattern<MouseButtonEventHandler, MouseButtonEventArgs>(h => uiElement.MouseLeftButtonDown += h, h => uiElement.MouseLeftButtonDown -= h)
.Select(o => DateTime.Now);
var mouseRightButtonActivity = Observable.FromEventPattern<MouseButtonEventHandler, MouseButtonEventArgs>(h => uiElement.MouseRightButtonDown += h, h => uiElement.MouseRightButtonDown -= h)
.Select(o => DateTime.Now);
var mouseWheelActivity = Observable.FromEventPattern<MouseWheelEventHandler, MouseWheelEventArgs>(h => uiElement.MouseWheel += h, h => uiElement.MouseWheel -= h)
.Select(o => DateTime.Now);
var keyboardActivity = Observable.FromEventPattern<KeyEventHandler, KeyEventArgs>(h => uiElement.KeyDown += h, h => uiElement.KeyDown -= h)
.Select(o => DateTime.Now);
var streetViewContainer = HtmlPage.Document.GetElementById("streetViewContainer");
var mouseMoveHandler = new EventHandler<HtmlEventArgs>(this.Moo);
bool b = streetViewContainer.AttachEvent("mousemove", mouseMoveHandler);
var browserActivity = Observable.FromEventPattern<Landmark.QDesk.ApplicationServices.IdleTimeoutService.MouseMoveHandler, HtmlEventArgs>(h => this.MyMouseMove += h, h => this.MyMouseMove -= h).Select(o => DateTime.Now);
// Merge the IObservables<DateTime> together into one stream/ sequence
this.allActivity = mouseMoveActivity.Merge(mouseLeftButtonActivity)
.Merge(mouseRightButtonActivity)
.Merge(mouseWheelActivity)
.Merge(keyboardActivity)
.Merge(browserActivity);
}
public IDisposable Subscribe(TimeSpan timeSpan, Action<DateTime> timeoutAction)
{
IObservable<DateTime> timeoutNotification = this.allActivity.Merge (IdleTimeoutService.GetDateTimeNowObservable())
.Throttle(timeSpan)
.ObserveOn(Scheduler.ThreadPool);
return timeoutNotification.Subscribe(timeoutAction);
}

There's an overload to Merge that takes in an IObservable<IObservable<TSource>>. Make the outer sequence a Subject<IObservable<TSource>> and call OnNext to it when you want to add another source to the bunch. The Merge operator will receive the source and subscribe to it:
var xss = new Subject<IObservable<int>>();
xss.Merge().Subscribe(x => Console.WriteLine(x));
xss.OnNext(Observable.Interval(TimeSpan.FromSeconds(1.0)).Select(x => 23 + 8 * (int)x));
xss.OnNext(Observable.Interval(TimeSpan.FromSeconds(0.8)).Select(x => 17 + 3 * (int)x));
xss.OnNext(Observable.Interval(TimeSpan.FromSeconds(1.3)).Select(x => 31 + 2 * (int)x));
...

The easiest way to do this would be to use an intermediate subject in place of the Merge calls.
Subject<DateTime> allActivities = new Subject<DateTime>();
var activitySubscriptions = new CompositeDisposable();
activitySubscriptions.Add(mouseMoveActivity.Subscribe(allActivities));
activitySubscriptions.Add(mouseLeftButtonActivity.Subscribe(allActivities));
//etc ...
//subscribe to activities
allActivities.Throttle(timeSpan)
.Subscribe(timeoutAction);
//later add another
activitySubscriptions.Add(newActivity.Subscribe(allActivities));
The Subject class will stop passing OnNext (and further OnError and OnCompleted) events from any of the observables it is subscribed to if it receives any OnError or OnCompleted.
The main difference between this approach and your sample is that it subscribes to all the events when the subject is created, rather than when you subscribe to the merged observable. Since all of the observables in your example are hot, the difference should not be noticeable.