Develop Reference

Reactive services. Grouping and caching streams

New: Entire source code with tests is now at https://github.com/bboyle1234/ReactiveTest
Let's imagine we have a view state object that is able to be updated by small partial view change events. Here are some example models of the total view, the incremental view update events and the accumulator function Update that builds the total view:
interface IDeviceView : ICloneable {
Guid DeviceId { get; }
}
class DeviceTotalView : IDeviceView {
public Guid DeviceId { get; set; }
public int Voltage { get; set; }
public int Currents { get; set; }
public object Clone() => this.MemberwiseClone();
}
class DeviceVoltagesUpdateView : IDeviceView {
public Guid DeviceId { get; set; }
public int Voltage { get; set; }
public object Clone() => this.MemberwiseClone();
}
class DeviceCurrentsUpdateView : IDeviceView {
public Guid DeviceId { get; set; }
public int Current { get; set; }
public object Clone() => this.MemberwiseClone();
}
class DeviceUpdateEvent {
public DeviceTotalView View;
public IDeviceView LastUpdate;
}
static DeviceUpdateEvent Update(DeviceUpdateEvent previousUpdate, IDeviceView update) {
if (update.DeviceId != previousUpdate.View.DeviceId) throw new InvalidOperationException("Device ids do not match (numskull exception).");
var view = (DeviceTotalView)previousUpdate.View.Clone();
switch (update) {
case DeviceVoltagesUpdateView x: {
view.Voltage = x.Voltage;
break;
}
case DeviceCurrentsUpdateView x: {
view.Currents = x.Current;
break;
}
}
return new DeviceUpdateEvent { View = view, LastUpdate = update };
}
Next, let's imagine we already have an injectable service that is able to produce an observable stream of the small update events for all devices, and that we want to create a service that can produce an aggregated view stream for individual devices.
Here is the interface of the service we want to create:
interface IDeviceService {
/// <summary>
/// Gets an observable that produces aggregated update events for the device with the given deviceId.
/// On subscription, the most recent event is immediately pushed to the subscriber.
/// There can be multiple subscribers.
/// </summary>
IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId);
}
How can I implement this interface and its requirements using the reactive extensions in the System.Reactive v4 library, targeting .netstandard2.0? Here's my boiler code with comments and that's as far as I've been able to get.
class DeviceService : IDeviceService {
readonly IObservable<IDeviceView> Source;
public DeviceService(IObservable<IDeviceView> source) { // injected parameter
/// When injected here, "source" is cold in the sense that it won't produce events until the first time it is subscribed.
/// "source" will throw an exception if its "Subscribe" method is called more than once as it is intended to have only one observer and
/// be read all the way from the beginning.
Source = source;
/// Callers of the "Subscribe" method below will expect data to be preloaded and will expect to be immediately delivered the most
/// recent event. So we need to immediately subscribe to "source" and start preloading the aggregate streams.
/// I'm assuming there is going to need to be a groupby to split the stream by device id.
var groups = source.GroupBy(x => x.DeviceId);
/// Now somehow we need to perform the aggregrate function on each grouping.
/// And create an observable that immediately delivers the most recent aggregated event when "Subscribe" is called below.
}
public IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId) {
/// How do we implement this? The observable that we return must be pre-loaded with the latest update
throw new NotImplementedException();
}
}

You have some weird code in that gist. Here's what I got working:
public class DeviceService : IDeviceService, IDisposable
{
readonly IObservable<IDeviceView> Source;
private readonly Dictionary<Guid, IObservable<DeviceUpdateEvent>> _updateStreams = new Dictionary<Guid, IObservable<DeviceUpdateEvent>>();
private readonly IObservable<(Guid, IObservable<DeviceUpdateEvent>)> _groupedStream;
private readonly CompositeDisposable _disposable = new CompositeDisposable();
public DeviceService(IObservable<IDeviceView> source)
{
Source = source;
_groupedStream = source
.GroupBy(v => v.DeviceId)
.Select(o => (o.Key, o
.Scan(new DeviceUpdateEvent { View = DeviceTotalView.GetInitialView(o.Key), LastUpdate = null }, (lastTotalView, newView) => lastTotalView.Update(newView))
.Replay(1)
.RefCount()
));
var groupSubscription = _groupedStream.Subscribe(t =>
{
_updateStreams[t.Item1] = t.Item2;
_disposable.Add(t.Item2.Subscribe());
});
_disposable.Add(groupSubscription);
}
public void Dispose()
{
_disposable.Dispose();
}
public IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId)
{
/// How do we implement this? The observable that we return must be pre-loaded with the latest update
if(this._updateStreams.ContainsKey(deviceId))
return this._updateStreams[deviceId];
return _groupedStream
.Where(t => t.Item1 == deviceId)
.Select(t => t.Item2)
.Switch();
}
}
The meat here is the _groupedStream piece. You group by DeviceId, as you said, then you use Scan to update state. I also moved Update to a static class and made it an extension method. You'll need an initial state, so I modified your DeviceTotalView class to get that. Modify accordingly:
public class DeviceTotalView : IDeviceView
{
public Guid DeviceId { get; set; }
public int Voltage { get; set; }
public int Currents { get; set; }
public object Clone() => this.MemberwiseClone();
public static DeviceTotalView GetInitialView(Guid deviceId)
{
return new DeviceTotalView
{
DeviceId = deviceId,
Voltage = 0,
Currents = 0
};
}
}
Next, the .Replay(1).Refcount() serves to remember the most recent update then provide that on subscription. We then stuff all of these child observables into a dictionary for easy retrieval on the method call. The dummy subscriptions (_disposable.Add(t.Item2.Subscribe())) are necessary for Replay to work.
In the event that there's an early request for a DeviceId that doesn't yet have an update, we subscribe to the _groupedStream which will wait for the first update, producing that Id's observable, then .Switch subscribes to that child observable.
However, all of this failed against your test code, I'm guessing because of the ConnectableObservableForAsyncProducerConsumerQueue class. I didn't want to debug that, because I wouldn't recommend doing something like that. In general it's not recommended to mix TPL and Rx code. They problems they solve largely overlap and they get in each other's way. So I modified your test code replacing that connectable observable queue thing with a Replay subject.
I also added the test-case for an early request (before an updates for that Device have arrived):
DeviceUpdateEvent deviceView1 = null;
DeviceUpdateEvent deviceView2 = null;
DeviceUpdateEvent deviceView3 = null;
var subject = new ReplaySubject<IDeviceView>();
var id1 = Guid.NewGuid();
var id2 = Guid.NewGuid();
var id3 = Guid.NewGuid();
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id1, Voltage = 1 });
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id1, Voltage = 2 });
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id2, Voltage = 100 });
var service = new DeviceService(subject);
service.GetDeviceStream(id1).Subscribe(x => deviceView1 = x);
service.GetDeviceStream(id2).Subscribe(x => deviceView2 = x);
service.GetDeviceStream(id3).Subscribe(x => deviceView3 = x);
/// I believe there is no need to pause here because the Subscribe method calls above
/// block until the events have all been pushed into the subscribers above.
Assert.AreEqual(deviceView1.View.DeviceId, id1);
Assert.AreEqual(deviceView2.View.DeviceId, id2);
Assert.AreEqual(deviceView1.View.Voltage, 2);
Assert.AreEqual(deviceView2.View.Voltage, 100);
Assert.IsNull(deviceView3);
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id2, Voltage = 101 });
Assert.AreEqual(deviceView2.View.Voltage, 101);
subject.OnNext(new DeviceVoltagesUpdateView { DeviceId = id3, Voltage = 101 });
Assert.AreEqual(deviceView3.View.DeviceId, id3);
Assert.AreEqual(deviceView3.View.Voltage, 101);
That passes fine and can be run without async.
Also, as a general tip, I would recommend doing unit tests for Rx code with the Microsoft.Reactive.Testing package, rather than time-gapping things.

A huge thanks to #Shlomo for the answer above.
The implementation given in the accepted answer, whilst a magical education for me, had a couple of issues that also needed to be solved in turn. The first was a threadrace problem, and the second was performance when a large number of devices were in the system. I ended up solving the threadrace AND dramatically improving performance with this modified implementation:
In the constructor, the grouped and scanned device stream is subscribed directly to a BehaviorSubject, which implements the Replay(1).RefCount() functionality required to immediately notify new subscribers of the latest value in the stream.
In the GetDeviceStream method, we continue to use a dictionary lookup to find the device stream, creating a preloaded BehaviorSubject if it doesn't already exist in the dictionary. We have removed the Where search that existed in the previous implementation in the question above. Using the where search caused a threadrace problem that was solved by making the grouped stream replayable. That caused an expontial performance issue. Replacing it with FirstOrDefault reduced the time take by half, and then removing it completely in favor of the GetCreate dictionary technique gave perfect perfomance O(1) instead of O(n2).
GetCreateSubject uses the Lazy proxy object as the dictionary value because the ConcurrentDictionary can sometimes call the Create method more than once for a single key. Supplying a Lazy to the dictionary ensures that the Value property is only called on one of the lazies, and therefore only one BehaviorSubject is created per device.
class DeviceService : IDeviceService, IDisposable {
readonly CompositeDisposable _disposable = new CompositeDisposable();
readonly ConcurrentDictionary<Guid, Lazy<BehaviorSubject<DeviceUpdateEvent>>> _streams = new ConcurrentDictionary<Guid, Lazy<BehaviorSubject<DeviceUpdateEvent>>>();
BehaviorSubject<DeviceUpdateEvent> GetCreateSubject(Guid deviceId) {
return _streams.GetOrAdd(deviceId, Create).Value;
Lazy<BehaviorSubject<DeviceUpdateEvent>> Create(Guid id) {
return new Lazy<BehaviorSubject<DeviceUpdateEvent>>(() => {
var subject = new BehaviorSubject<DeviceUpdateEvent>(DeviceUpdateEvent.GetInitialView(deviceId));
_disposable.Add(subject);
return subject;
});
}
}
public DeviceService(IConnectableObservable<IDeviceView> source) {
_disposable.Add(source
.GroupBy(x => x.DeviceId)
.Subscribe(deviceStream => {
_disposable.Add(deviceStream
.Scan(DeviceUpdateEvent.GetInitialView(deviceStream.Key), DeviceUtils.Update)
.Subscribe(GetCreateSubject(deviceStream.Key)));
}));
_disposable.Add(source.Connect());
}
public void Dispose() {
_disposable.Dispose();
}
public IObservable<DeviceUpdateEvent> GetDeviceStream(Guid deviceId) {
return GetCreateSubject(deviceId).AsObservable();
}
}
[TestMethod]
public async Task Test2() {
var input = new AsyncProducerConsumerQueue<IDeviceView>();
var source = new ConnectableObservableForAsyncProducerConsumerQueue<IDeviceView>(input);
var service = new DeviceService(source);
var ids = Enumerable.Range(0, 100000).Select(i => Guid.NewGuid()).ToArray();
var idsRemaining = ids.ToHashSet();
var t1 = Task.Run(async () => {
foreach (var id in ids) {
await input.EnqueueAsync(new DeviceVoltagesUpdateView { DeviceId = id, Voltage = 1 });
}
});
var t2 = Task.Run(() => {
foreach (var id in ids) {
service.GetDeviceStream(id).Subscribe(x => idsRemaining.Remove(x.View.DeviceId));
}
});
await Task.WhenAll(t1, t2);
var sw = Stopwatch.StartNew();
while (idsRemaining.Count > 0) {
if (sw.Elapsed.TotalSeconds > 600) throw new Exception("Failed");
await Task.Delay(100);
}
}
See entire problem source code and test code here: https://github.com/bboyle1234/ReactiveTest

Return error from models

I have the following model:
internal static List<Contracts.DataContracts.Report> GetReportsForSearch(string searchVal, string searchParam)
{
var param1 = new SqlParameter("#SearchVal", searchVal);
var ctx = new StradaDataReviewContext2();
var reports = new List<Contracts.DataContracts.Report>();
try
{
//Validate param1 here and return false if the requirment are not met
}
catch(Exception e)
{
//Throw
}
}
param1 here Is a value entered by a user and I want to validate It here, and If the requirements are not met, I want to return an error.
But how can I return an error here from the model? The method Is of the type List, and I can't not just write return false in this method.
Any suggestion how to do It?

It is good that you didn't thought about throwing an exception, when requirements are not met. We shouldn't use exceptions for controlling program flow.
I have two options in my mind :
1. Use objects
Modify your GetReportsForSearch method to following signature:
internal static List<Contracts.DataContracts.Report> GetReportsForSearch(string searchVal,
string searchParam, ReportRequestor requestor)
{
var param1 = new SqlParameter("#SearchVal", searchVal);
var ctx = new StradaDataReviewContext2();
var reports = new List<Contracts.DataContracts.Report>();
try
{
//Validate param1 here and call RequirementsAreNotMet method if the requirements are not met
requestor.RequirementsAreNotMet();
}
catch(Exception e)
{
//Throw
}
}
And then you can implement code responsible for handling this situation in ReportRequestor class
public class ReportRequestor
{
public void RequiremenrsAreNotMet()
{
//code which handle situation when requiremenets are not met
}
}
2. Use return type as indicator of status
In this way, when requirements are not met you should create ReportGenerationStatus object with HasResult flag set to false.
In other case just set HasResult to true and also set results accordingly. This somewhat mimics Option type known from functional languages
internal static ReportGenerationStatus GetReportsForSearch(string searchVal, string searchParam)
{
//code for your method
}
public class ReportGenerationStatus
{
public List<Contracts.DataContracts.Report> Result { get; set; }
public bool HasResult { get; set; }
}

Catch exceptions in async loading of dialog viewmodel

I have a DialogViewModel class with async Task LoadData() method. This method loads data asynchronously and shows this dialog, which notifies user about loading. Here is the code:
try
{
var dialog = new DialogViewModel();
var loadTask = dialog.LoadData();
WindowManager.ShowDialog(dialog);
await loadTask;
}
catch (Exception ex)
{
Logger.Error("Error in DialogViewModel", ex);
// Notify user about the error
}
When LoadData throws an exception, it isn't handled until user exits the dialog. It happens because exception is handled when calling await, and it's not happening until WindowManager.ShowDialog(dialog) completes.
What is the correct way to show a dialog with async loading? I've tried this ways:
Call LoadData() in OnShow(), constructor or similar. But this won't work if I'll need to show this dialog without any data
Call await LoadData() before showing the dialog. This way user have to wait for data to load before actually seeing the window, but I want the window to show up instantly with a loading indicator.

Why is there an explicit public LoadData method?
If this has to happen then do it inside the constructor asynchronously using Task<T> with a ContinueWith to process any exception generated by checking the IsFaultedproperty on the returned task.
This would address both issues you've highlighted.
A very simple example is shown below, obivously you're implementation will be more complicated.
public class DialogViewModel
{
private Task _task;
public DialogViewModel()
{
var context = TaskScheduler.FromCurrentSynchronizationContext();
_task = Task.Factory.StartNew(() =>
{
var data = GetDataCollection();
return data;
})
.ContinueWith(t =>
{
if (t.IsFaulted)
{
HasErrored = true;
ErrorMessage = "It's borked!";
}
else
{
Data = t.Result;
}
}, context);
}
public IEnumerable<string> Data { get; private set; }
public bool HasErrored { get; private set; }
public string ErrorMessage { get; private set; }
private static IEnumerable<string> GetDataCollection()
{
return new List<string>()
{
"John",
"Jack",
"Steve"
};
}
}
Or if you don't want to use Task<T> explicitly and want to use async\await functionality you could use a slightly different approach because you can't use async\await with a class constructor:
public class DialogViewModel
{
public IEnumerable<string> Data { get; private set; }
public bool HasErrored { get; private set; }
public string ErrorMessage { get; private set; }
async public static Task<DialogViewModel> BuildViewModelAsync()
{
try
{
var data = await GetDataCollection();
return new DialogViewModel(data);
}
catch (Exception)
{
return new DialogViewModel("Failed!");
}
}
private DialogViewModel(IEnumerable<string> data)
{
Data = data;
}
private DialogViewModel(string errorMessage)
{
HasErrored = true;
ErrorMessage = errorMessage;
}
private async static Task<IEnumerable<string>> GetDataCollection()
{
// do something async...
return await Task.Factory.StartNew(() => new List<string>()
{
"John",
"Jack",
"Steve"
});
}
}

Cannot return IObservable<T> from a method marked async

This contrived example is roughly how my code is structured:
public abstract class SuperHeroBase
{
protected SuperHeroBase() { }
public async Task<CrimeFightingResult> FightCrimeAsync()
{
var result = new CrimeFightingResult();
result.State = CrimeFightingStates.Fighting;
try
{
await FightCrimeOverride(results);
}
catch
{
SetError(results);
}
if (result.State == CrimeFightingStates.Fighting)
result.State = CrimeFightingStates.GoodGuyWon;
return result;
}
protected SetError(CrimeFightingResult results)
{
result.State = CrimeFightingStates.BadGuyWon;
}
protected abstract Task FightCrimeOverride(CrimeFightingResult results);
}
public enum CrimeFightingStates
{
NotStarted,
Fighting,
GoodGuyWon, // success state
BadGuyWon // error state
}
public class CrimeFightingResult
{
internal class CrimeFightingResult() { }
public CrimeFightingStates State { get; internal set; }
}
Now I'm trying to build a collection that would hold multiple SuperHero objects and offer a AllHerosFightCrime method. The hero's should not all fight at once (the next one starts when the first is finished).
public class SuperHeroCollection : ObservableCollection<SuperHeroBase>
{
public SuperHeroCollection() { }
// I mark the method async...
public async IObservable<CrimeFightingResult> AllHerosFightCrime()
{
var heros = new List<SuperHeroBase>(this);
var results = new ReplaySubject<CrimeFightingResult>();
foreach (var hero in heros)
{
// ... so I can await on FightCrimeAsync and push
// the result to the subject when done
var result = await hero.FightCrimeAsync();
results.OnNext(result);
}
results.OnCompleted();
// I can't return the IObservable here because the method is marked Async.
// It expects a return type of CrimeFightingResult
return results;
}
}
How can I return the IObservable<CrimeFightingResults> and still have the call to FightCrimeAsync awaited?

You could turn your task into an observable and combine them using Merge:
public IObservable<CrimeFightingResult> AllHerosFightCrime()
{
var heros = new List<SuperHeroBase>(this);
return heros.Select(h => h.FightCrimeAsync().ToObservable())
.Merge();
}
If you want to maintain the order your events are received you can use Concat instead of Merge.

Error Reporting

I have a class that follows the Command Pattern.
It has 2 methods which are Execute, and CanExecute which checks whether to invoke Execute or not (they derive from ICommand).
CanExecute invokes a few methods that check that all required services are running, the version is correct, etc.
After CanExecute is invoked, it may fail and return false and I need to know why. Is it because of a bad version, services, missing file, etc.
What is the best strategy to know what is the problem
One option is whenever a required condition fails I can throw an exception that will describe the error in the message field. However the possibility that it will fail is expected and you shouldn't use exceptions for regular flow of control. So I'm really not sure.
Thank you.

You can use a collection of "reasons" that will tell the users of the class why CanExecute returned false. The reasons can be a simple IEnumerable<string>.
public bool CanExecute() {
var messages = new List<string>();
if (!Condition1) {
messages.Add("Missing Condition1");
}
...
Messages = messages;
return messages.Count == 0;
}
public IEnumerable<string> Messages { get; private set; }
Then, client code can show the collection of messages to end-users.
UPDATE:
You can also associate new commands with the messages to give the users ways to fix the problems found. In this case, instead of an IEnumerable<string>, you can create your own class that encapsulates that information:
public class Message {
public string Text { get; set; }
public ICommand Command { get; set; }
}
...
public bool CanExecute() {
var messages = new List<Message>();
if (!Condition1) {
messages.Add(
new Message {
Text = "Missing Condition1",
Command = new FixCondition1Command()
}
);
}
...
Messages = messages;
return messages.Count == 0;
}
public IEnumerable<Message> Messages { get; private set; }

UPDATE: Reworked based on feedback.
Since the UI needs the reasons CanExecute() returns false, two things come to mind:
Option 1: Add an enumerable message property to the command interface and populate it as needed during the call to CanExecute(). The UI could then interrogate the property as needed. If you go this route, make sure you clear out the contents of the property each call to CanExecute() so you don't lose track of state.
public interface ICommand
{
IEnumerable<string> Messages { get; }
bool CanExecute();
void Execute();
}
public class SomeCommand : ICommand
{
public IEnumerable<string> Messages { get; private set; }
public bool CanExecute()
{
var messages = new List<string>();
var canExecute = true;
if (SomeCondition)
{
canExecute = false;
messages.Add("Some reason");
}
if (AnotherCondition)
{
canExecute = false;
messages.Add("Another reason");
}
Messages = messages;
return canExecute;
}
public void Execute() { }
}
Option 2: Have CanExecute() return an object which contains the bool as well as an enumerable messages property. This makes it obvious that the messages only apply to that call of CanExecute(). However, depending on where/how you're implementing (e.g. data binding), this could complicate other scenarios more than you're looking for.
public class CanExecuteResult
{
public bool CanExecute { get; set; }
public IEnumerable<string> Messages { get; set; }
}
public interface ICommand
{
CanExecuteResult CanExecute();
void Execute();
}
public class SomeCommand : ICommand
{
public CanExecuteResult CanExecute()
{
var result = new CanExecuteResult { CanExecute = true };
var messages = new List<string>();
if (SomeCondition)
{
result.CanExecute = false;
messages.Add("Some reason");
}
if (AnotherCondition)
{
result.CanExecute = false;
messages.Add("Another reason");
}
result.Messages = messages;
return result;
}
public void Execute() { }
}
Obviously, the specifics of how you want to handle the interfaces, enumerable types, etc. is up to you. The code is just a representation of the idea.

Bool CanExecute()
{
if(!CheckXXX)
throw new Exception("CheckXXX function throws an exception")
if(!CheckYYY)
throw new Exception("CheckYYY function throws an exception")
if(!CheckZZZ)
throw new Exception("CheckZZZ function throws an exception")
return true; //everything is working fine
}