Develop Reference

How to run parallel code for multiple search methods

I'm working on a code I would like to improve. It is a search method. Based on an input I would like to search this value in multiple tables of my database.
public async Task<IEnumerable<SearchResponseModel>> Search(string input)
{
var listOfSearchResponse = new List<SearchResponseModel>();
listOfSearchResponse.AddRange(await SearchOrder(input)),
listOfSearchResponse.AddRange(await SearchJob(input));
listOfSearchResponse.AddRange(await SearchClient(input));
listOfSearchResponse.AddRange(await SearchItem(input));
listOfSearchResponse.AddRange(await SearchProduction(input));
return listOfSearchResponse;
}
I use the work await because every search is defined like this one:
public async Task<IEnumerable<SearchResponseModel>> SearchOrder(string input) {...}
My five search methods are not yet really async. They all execute in sequence after the previous one. What should I do from here to make them parallel?

I would think that something like this should work, in theory:
var tasks = new[]
{
SearchOrder(input),
SearchJob(input),
SearchClient(input),
SearchItem(input),
SearchProduction(input)
};
await Task.WhenAll(tasks);
//var listOfSearchResponse = tasks.Select(t => t.Result).ToList();
var listOfSearchResponse = tasks. SelectMany(t => t.Result).ToList();
In practice, it's hard to know how much benefit you'll see.

It's worth considering using Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:
public IObservable<SearchResponseModel> SearchObservable(string input) =>
Observable.Defer<SearchResponseModel>(() =>
new []
{
Observable.FromAsync(() => SearchOrder(input)),
Observable.FromAsync(() => SearchJob(input)),
Observable.FromAsync(() => SearchClient(input)),
Observable.FromAsync(() => SearchItem(input)),
Observable.FromAsync(() => SearchProduction(input)),
}
.Merge()
.SelectMany(x => x));
The advantage here is that as each search completes you get the partial results through from the observable - there's no need to wait until all the tasks have finished.
Observables signal each value as they are produced and they signal a finial completion so you know when all of the results are through.

rx.net locking up from use of ToEnumerable

I am trying to convert the below statement so that I can get the key alongside the selected list:
var feed = new Subject<TradeExecuted>();
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(TimeSpan.FromSeconds(5)))
.SelectMany(x => x.ToList())
.Select(trades => Observable.FromAsync(() => Mediator.Publish(trades, cts.Token)))
.Concat() // Ensure that the results are serialized.
.Subscribe(cts.Token); // Check status of calls.
The above works, whereas the below does not - when I try and itterate over the list, it locks up.
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(timespan))
.Select(x => Observable.FromAsync(() =>
{
var list = x.ToEnumerable(); // <---- LOCK UP if we use list.First() etc
var aggregate = AggregateTrades(x.Key.Symbol, x.Key.AccountId, x.Key.Tenant, list);
return Mediator.Publish(aggregate, cts.Token);
}))
.Concat()
.Subscribe(cts.Token); // Check status of calls.
I am clearly doing something wrong and probably horrific!
Going back to the original code, how can I get the Key alongside the enumerable list (and avoiding the hack below)?
As a sidenote, the below code works but it a nasty hack where I get the keys from the first list item:
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(TimeSpan.FromSeconds(5)))
.SelectMany(x => x.ToList())
.Select(trades => Observable.FromAsync(() =>
{
var firstTrade = trades.First();
var aggregate = AggregateTrades(firstTrade.Execution.Contract.Symbol, firstTrade.Execution.AccountId, firstTrade.Tenant, trades);
return Mediator.Publish(aggregate, cts.Token);
}))
.Concat() // Ensure that the results are serialized.
.Subscribe(cts.Token); // Check status of calls.

All versions of your code suffer from trying to eagerly evaluate the grouped sub-observable. Since in v1 and v3 your group observable will run a maximum of 5 seconds, that isn't horrible/awful, but it's still not great. In v2, I don't know what timespan is, but assuming it's 5 seconds, you have the same problem: Trying to turn the grouped sub-observable into a list or an enumerable means waiting for the sub-observable to complete, blocking the thread (or the task).
You can fix this by using the Buffer operator to lazily evaluate the grouped sub-observable:
var timespan = TimeSpan.FromSeconds(5);
feed
.GroupByUntil(x => (x.Execution.Contract.Symbol, x.Execution.AccountId, x.Tenant, x.UserId), x => Observable.Timer(timespan))
.SelectMany(x => x
.Buffer(timespan)
.Select(list => Observable.FromAsync(() =>
{
var aggregate = AggregateTrades(x.Key.Symbol, x.Key.AccountId, x.Key.Tenant, list));
return Mediator.Publish(aggregate, cts.Token);
}))
)
.Concat() // Ensure that the results are serialized.
.Subscribe(cts.Token); // Check status of calls.
This essentially means that until timespan is up, the items in the group by gather in a list inside Buffer. Once timespan is up, they're released as a list, and the mediator publish happens.

Observe values not seen in other observers

I have an observable that emits unique values e.g.
var source=Observable.Range(1,100).Publish();
source.Connect();
I want to observe its values from e.g. two observers but each observer to get notified only for values not seen in other observers.
So if first observer contains the value 10 the second observer should never get notified for the 10 value.
Update
I chose #Asti`s answer cause it was first and although buggy it pointed to the right direction and up-voted #Shlomo's answer. Too bad I cannot accept both answers as #Shlomo answer was more correct and I really appreciate all his help we get on this tag.

Observables aren't supposed to behave differently for different observers; a better approach would be to give each observer its own filtered observable.
That being said, if your constraints require that you need this behavior in a single observable - we can use a Round-Robin method.
public static IEnumerable<T> Repeat<T>(this IEnumerable<T> source)
{
for (; ; )
foreach (var item in source.ToArray())
yield return item;
}
public static IObservable<T> RoundRobin<T>(this IObservable<T> source)
{
var subscribers = new List<IObserver<T>>();
var shared = source
.Zip(subscribers.Repeat(), (value, observer) => (value, observer))
.Publish()
.RefCount();
return Observable.Create<T>(observer =>
{
subscribers.Add(observer);
var subscription =
shared
.Where(pair => pair.observer == observer)
.Select(pair => pair.value)
.Subscribe(observer);
var dispose = Disposable.Create(() => subscribers.Remove(observer));
return new CompositeDisposable(subscription, dispose);
});
}
Usage:
var source = Observable.Range(1, 100).Publish();
var dist = source.RoundRobin();
dist.Subscribe(i => Console.WriteLine($"One sees {i}"));
dist.Subscribe(i => Console.WriteLine($"Two sees {i}"));
source.Connect();
Result:
One sees 1
Two sees 2
One sees 3
Two sees 4
One sees 5
Two sees 6
One sees 7
Two sees 8
One sees 9
Two sees 10
If you already have a list of observers, the code becomes much simpler.

EDIT: #Asti fixed his bug, and I fixed mine based on his answer. Our answers are now largely similar. I have an idea how to do a purely reactive one, if I have time I'll post that later.
Fixed code:
public static IObservable<T> RoundRobin2<T>(this IObservable<T> source)
{
var subscribers = new BehaviorSubject<ImmutableList<IObserver<T>>>(ImmutableList<IObserver<T>>.Empty);
ImmutableList<IObserver<T>> latest = ImmutableList<IObserver<T>>.Empty;
subscribers.Subscribe(l => latest = l);
var shared = source
.Select((v, i) => (v, i))
.WithLatestFrom(subscribers, (t, s) => (t.v, t.i, s))
.Publish()
.RefCount();
return Observable.Create<T>(observer =>
{
subscribers.OnNext(latest.Add(observer));
var dispose = Disposable.Create(() => subscribers.OnNext(latest.Remove(observer)));
var sub = shared
.Where(t => t.i % t.s.Count == t.s.FindIndex(o => o == observer))
.Select(t => t.v)
.Subscribe(observer);
return new CompositeDisposable(dispose, sub);
});
}
Original answer:
I upvoted #Asti's answer, because he's largely correct: Just because you can, doesn't mean you should. And his answer largely works, but it's subject to a bug:
This works fine:
var source = Observable.Range(1, 20).Publish();
var dist = source.RoundRobin();
dist.Subscribe(i => Console.WriteLine($"One sees {i}"));
dist.Take(1).Subscribe(i => Console.WriteLine($"Two sees {i}"));
This doesn't:
var source = Observable.Range(1, 20).Publish();
var dist = source.RoundRobin();
dist.Take(1).Subscribe(i => Console.WriteLine($"One sees {i}"));
dist.Subscribe(i => Console.WriteLine($"Two sees {i}"));
Output is:
One sees 1
Two sees 1
Two sees 2
Two sees 3
Two sees 4
...
I first thought the bug is Halloween related, but now I'm not sure. The .ToArray() in Repeat should take care of that. I also wrote a pure-ish observable implementation which has the same bug. This implementation doesn't guarantee a perfect Round Robin, but that wasn't in the question:
public static IObservable<T> RoundRobin2<T>(this IObservable<T> source)
{
var subscribers = new BehaviorSubject<ImmutableList<IObserver<T>>>(ImmutableList<IObserver<T>>.Empty);
ImmutableList<IObserver<T>> latest = ImmutableList<IObserver<T>>.Empty;
subscribers.Subscribe(l => latest = l);
var shared = source
.Select((v, i) => (v, i))
.WithLatestFrom(subscribers, (t, s) => (t.v, t.i, s))
.Publish()
.RefCount();
return Observable.Create<T>(observer =>
{
subscribers.OnNext(latest.Add(observer));
var dispose = Disposable.Create(() => subscribers.OnNext(latest.Remove(observer)));
var sub = shared
.Where(t => t.i % t.s.Count == t.s.FindIndex(o => o == observer))
.Select(t => t.v)
.Subscribe(observer);
return new CompositeDisposable(dispose, sub);
});
}

This is a simple distributed queue implementation using TPL Dataflow. But with respect to different observers not seeing the same value, there's little chance of it behaving incorrectly. It's not round-robin, but actually has back-pressure semantics.
public static IObservable<T> Distribute<T>(this IObservable<T> source)
{
var buffer = new BufferBlock<T>();
source.Subscribe(buffer.AsObserver());
return Observable.Create<T>(observer =>
buffer.LinkTo(new ActionBlock<T>(observer.OnNext, new ExecutionDataflowBlockOptions { BoundedCapacity = 1 })
);
}
Output
One sees 1
Two sees 2
One sees 3
Two sees 4
One sees 5
One sees 6
One sees 7
One sees 8
One sees 9
One sees 10
I might prefer skipping Rx entirely and just using TPL Dataflow.

Run processing in separated thread by group

I'm trying to use Rx in my Kafka consumer.
public static event EventHandler<ConsumeResult<string, string>> GenericEvent;
then I have the following code
var observable = Observable.FromEventPattern<ConsumeResult<string, string>>(
x => GenericEvent += x,
x => GenericEvent -= x)
.Select(x => x.EventArgs);
while (!cancellationToken.IsCancellationRequested)
{
ConsumeResult<string, string> consumeResult = _consumer.Consume(cancellationToken);
GenericEvent(consumeResult.Topic, consumeResult);
}
then somewhere I use it like
var subscription = observable.Subscribe(message =>
{
Console.WriteLine($"{Thread.CurrentThread.ManagedThreadId} ** {message.Topic}/{message.Partition} #{message.Offset}: '{message.Value}'");
//_kafkaConsumer.Commit(messages);
});
Are the any possibility to run separated thread by topic name (consumeResult.Topic)? When consumer receive a message, it should redirect it to the corresponding thread by topic

Give this a go:
Observable
.Interval(TimeSpan.FromSeconds(0.1))
.Take(20)
.GroupBy(x => x % 3)
.SelectMany(xs => Observable.Using(() => new EventLoopScheduler(), els => xs.ObserveOn(els)))
.Subscribe(x => Console.WriteLine($"{x} {Thread.CurrentThread.ManagedThreadId}"));
This ensures that a thread is created in the new EventLoopScheduler() scheduler for each inner observable created by the GroupBy operator. The SelectMany flattens out the group, but keeps the EventLoopScheduler associated for each group.
In your case you GroupBy the consumeResult.Topic property.
Do make sure that your source observable ends as the threads live forever until they do. Calling Dispose() on the subscription is sufficient to end the observable.

GroupBy with latest element and updates instead of key and updates

I got an Observable, I want to group them modulo 10. I want to make the result to be an hot observable and that when a new subscriber subscribes he got all of the GroupedObservable being played before but instead of getting the Key, I want the latest value. I want the updates as well, skipping the latest value.
In this example, to make it simpler, we will only work with modulo result equals 5. But I want my solution working for everything.
Example:
--------15-----25---------...
Let's add on the previous example some points in time where we will subscribe:
---S1-----15--S2---25-----S3----...
The expected results are:
S1 to receive Latest: 15, Updates: an observable starting with 25 and updates %10 ==5 that will arrive later.
Explanation: S1 will be notified after the 15 arrives, 15 is the latest element so I want it straight away. The second argument will be an observable that will produce later 25 and the %10 == 5 elements in the future.
S2 to receive Latest: 15, Updates: an observable starting with 25 and updates %10 ==5 that will arrive later.
Explanation: S2 will be notified on subscription, 15 is the latest element so I want it straight away. The second argument will be an observable that will produce later 25 and the %10 == 5 elements in the future.
S3 to receive Latest: 25, Updates: an observable with updates %10 ==5 that will arrive later.
Explanation: S3 will be notified on subscription, 25 is the latest element so I want it straight away. The second argument will be an observable that will produce the %10 == 5 elements in the future.
Here are some resolution tries:
The code below use Tuple and NUnit.
First attempt
[Test]
public void WhenWeGroupByReplaying1()
{
var subject = new Subject<uint>();
var observable = subject.GroupBy(t => t%10)
.Select(t =>
{
var connectableObservable = t.Replay(1);
connectableObservable.Connect();
return (key: t.Key, updates: connectableObservable);
}).Replay();
observable.Connect();
// I will block on the First of the lambda below
var getLastAndUpdates = observable
.Select(t => (first: t.updates.First(),updates: t.updates.Skip(1)));
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[1] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[1] - UPDATE: {t2}"));
});
subject.OnNext(15);
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[2] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[2] - UPDATE: {t2}"));
});
subject.OnNext(25);
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[3] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[3] - UPDATE: {t2}"));
});
}
This solution will be blocking as shown in comment.
Second attempt
[Test]
public void WhenWeGroupByReplaying2()
{
var subject = new Subject<uint>();
var observable = subject.GroupBy(t => t, t => t, new ModuloEqualityComparer())
.Select(t =>
{
var connectableObservable = t.Publish(t.Key);
connectableObservable.Connect();
return (key: t.Key, updates: connectableObservable);
}).Replay();
observable.Connect();
var getLastAndUpdates = observable
.Select(t => (first: t.updates.First(),updates: t.updates.Skip(1)));
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[1] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[1] - UPDATE: {t2}"));
});
subject.OnNext(15);
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[2] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[2] - UPDATE: {t2}"));
});
subject.OnNext(25);
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[3] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[3] - UPDATE: {t2}"));
});
}
private class ModuloEqualityComparer : IEqualityComparer<uint>
{
public bool Equals(uint x, uint y)
{
return x % 10 == y % 10;
}
public int GetHashCode(uint obj)
{
return (obj % 10).GetHashCode();
}
}
Result:
[1] - FIRST: 15
[1] - UPDATE: 15
[2] - FIRST: 15
[1] - UPDATE: 25
[2] - UPDATE: 25
[3] - FIRST: 25
Expected result: (exact order doesn't mater)
[1] - FIRST: 15
[2] - FIRST: 15
[1] - UPDATE: 25
[2] - UPDATE: 25
[3] - FIRST: 25
Third attempt
[Test]
public void WhenWeGroupByReplaying3()
{
var subject = new Subject<uint>();
var observable = subject.GroupBy(t => (key: t%10, value:t), t => t, new ModuloEqualityComparer2())
.Select(t =>
{
var connectableObservable = t.Publish(t.Key.Item2);
connectableObservable.Connect();
return (key: t.Key, updates: connectableObservable);
}).Replay();
observable.Connect();
var getLastAndUpdates = observable
.Select(t => (first: t.updates.First(),updates: t.updates.Skip(1)));
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[1] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[1] - UPDATE: {t2}"));
});
subject.OnNext(15);
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[2] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[2] - UPDATE: {t2}"));
});
subject.OnNext(25);
getLastAndUpdates.Subscribe(t =>
{
Console.WriteLine($"[3] - FIRST: {t.first}");
t.updates.Subscribe(t2 => Console.WriteLine($"[3] - UPDATE: {t2}"));
});
}
private class ModuloEqualityComparer2 : IEqualityComparer<(uint,uint)>
{
private readonly ModuloEqualityComparer _moduloEqualityComparer = new ModuloEqualityComparer();
public bool Equals((uint, uint) x, (uint, uint) y)
{
return _moduloEqualityComparer.Equals(x.Item1, y.Item1);
}
public int GetHashCode((uint, uint) obj)
{
return _moduloEqualityComparer.GetHashCode(obj.Item1);
}
}
Result:
[1] - FIRST: 15
[1] - UPDATE: 15
[2] - FIRST: 15
[1] - UPDATE: 25
[2] - UPDATE: 25
[3] - FIRST: 25
Expected result: (exact order doesn't mater)
[1] - FIRST: 15
[2] - FIRST: 15
[1] - UPDATE: 25
[2] - UPDATE: 25
[3] - FIRST: 25
Thanks for reading.

I'm not entirely sure what you're trying to achieve, but hopefully this will help you:
There's a couple things wrong with your code:
.First() is obsolete for a reason. You shouldn't be using blocking code with Rx
.Replay() requires a dummy subscription to work correctly. I'm not sure if that's what was plaguing your code or not, but to achieve your aims, you want that.
Nested subscriptions are generally a bad idea. I have replaced the nested subscriptions with a .Merge().
If this doesn't solve your problem, I suggest amending your question to describe what you're trying to accomplish by using Rx. This smells a bit like an XY situation.
Here's the code:
var subject = new Subject<uint>();
var observable = subject.GroupBy(t => t % 10)
.Select(t => t.Replay(1).RefCount()).Replay().RefCount();
// dummy subscriptions required for Replay to work correctly.
var dummySub = observable.Merge().Subscribe();
observable
.Select(o => o.Select((t, index) => (t.Key, t.num, index)))
.Merge()
.Subscribe(t =>
{
if (t.index == 0)
Console.WriteLine($"[1] - FIRST: {t.num}");
else
Console.WriteLine($"[1] - UPDATE: {t.num}");
});
subject.OnNext(15);
observable
.Select(o => o.Select((t, index) => (t.Key, t.num, index)))
.Merge()
.Subscribe(t =>
{
if (t.index == 0)
Console.WriteLine($"[1] - FIRST: {t.num}");
else
Console.WriteLine($"[1] - UPDATE: {t.num}");
});
subject.OnNext(25);
observable
.Select(o => o.Select((t, index) => (t.Key, t.num, index)))
.Merge()
.Subscribe(t =>
{
if (t.index == 0)
Console.WriteLine($"[1] - FIRST: {t.num}");
else
Console.WriteLine($"[1] - UPDATE: {t.num}");
});