Develop Reference

Rx produce and consume on different threads

I have tried to simplify my issue by a sample code here. I have a producer thread constantly pumping in data and I am trying to batch it with a time delay between batches so that the UI has time to render it. But the result is not as expected, the produce and consumer seems to be on the same thread.
I don't want the batch buffer to sleep on the thread that is producing. Tried SubscribeOn did not help much. What am I doing wrong here, how do I get this to print different thread Ids on producer and consumer thread.
static void Main(string[] args)
{
var stream = new ReplaySubject<int>();
Task.Factory.StartNew(() =>
{
int seed = 1;
while (true)
{
Console.WriteLine("Thread {0} Producing {1}",
Thread.CurrentThread.ManagedThreadId, seed);
stream.OnNext(seed);
seed++;
Thread.Sleep(TimeSpan.FromMilliseconds(500));
}
});
stream.Buffer(5).Do(x =>
{
Console.WriteLine("Thread {0} sleeping to create time gap between batches",
Thread.CurrentThread.ManagedThreadId);
Thread.Sleep(TimeSpan.FromSeconds(2));
})
.SubscribeOn(NewThreadScheduler.Default).Subscribe(items =>
{
foreach (var item in items)
{
Console.WriteLine("Thread {0} Consuming {1}",
Thread.CurrentThread.ManagedThreadId, item);
}
});
Console.Read();
}

Understanding the difference between ObserveOn and SubscribeOn is key here. See - ObserveOn and SubscribeOn - where the work is being done for an in depth explanation of these.
Also, you absolutely don't want to use a Thread.Sleep in your Rx. Or anywhere. Ever. Do is almost as evil, but Thead.Sleep is almost always totally evil. Buffer has serveral overloads you want to use instead - these include a time based overload and an overload that accepts a count limit and a time-limit, returning a buffer when either of these are reached. A time-based buffering will introduce the necessary concurrency between producer and consumer - that is, deliver the buffer to it's subscriber on a separate thread from the producer.
Also see these questions and answers which have good discussions on keeping consumers responsive (in the context of WPF here, but the points are generally applicable).
Process lots of small tasks and keep the UI responsive
Buffer data from database cursor while keeping UI responsive
The last question above specifically uses the time-based buffer overload. As I said, using Buffer or ObserveOn in your call chain will allow you to add concurrency between producer and consumer. You still need to take care that the processing of a buffer is still fast enough that you don't get a queue building up on the buffer subscriber.
If queues do build up, you'll need to think about means of applying backpressure, dropping updates and/or conflating the updates. These is a big topic too broad for in depth discussion here - but basically you either:
Drop events. There have been many ways discussed to tackle this in Rx. I current like Ignore incoming stream updates if last callback hasn't finished yet but also see With Rx, how do I ignore all-except-the-latest value when my Subscribe method is running and there are many other discussions of this.
Signal the producer out of band to tell it to slow down or send conflated updates, or
You introduce an operator that does in-stream conflation - like a smarter Buffer that could compress events to, for example, only include the latest price on a stock item etc. You can author operators that are sensitive to the time that OnNext invocations take to process, for example.
See if proper buffering helps first, then think about throttling/conflating events at the source as (a UI can only show so much infomation anway) - then consider smarter conflation as this can get quite complex. https://github.com/AdaptiveConsulting/ReactiveTrader is a good example of a project using some advanced conflation techniques.

Although the other answers are correct, I'd like to identify your actual problem as perhaps a misunderstanding of the behavior of Rx. Putting the producer to sleep blocks subsequent calls to OnNext and it seems as though you're assuming Rx automatically calls OnNext concurrently, but in fact it doesn't for very good reasons. Actually, Rx has a contract that requires serialized notifications.
See §§4.2, 6.7 in the Rx Design Guidelines for details.
Ultimately, it looks as though you're trying to implement the BufferIntrospective operator from Rxx. This operator allows you to pass in a concurrency-introducing scheduler, similar to ObserveOn, to create a concurrency boundary between a producer and a consumer. BufferIntrospective is a dynamic backpressure strategy that pushes out heterogeneously-sized batches based on the changing latencies of an observer. While the observer is processing the current batch, the operator buffers all incoming concurrent notifications. To accomplish this, the operator takes advantage of the fact that OnNext is a blocking call (per the §4.2 contract) and for that reason this operator should be applied as close to the edge of the query as possible, generally immediately before you call Subscribe.
As James described, you could call it a "smart buffering" strategy itself, or see it as the baseline for implementing such a strategy; e.g., I've also defined a SampleIntrospective operator that drops all but the last notification in each batch.

ObserveOn is probably what you want. It takes a SynchronizationContext as an argument, that should be the SynchronizationContext of your UI. If you don't know how to get it, see Using SynchronizationContext for sending events back to the UI for WinForms or WPF

What is the correct way to schedule periodic events in Rx?

Simple question, I would hope: I'm writing an application in which I want to retrieve data from a database; I've elected to use Rx for this purpose to represent the database as a sequence of values.
I only want to poll the database (and thus have my observer's notifications occur) at a maximum of once every 5 seconds. Right now, I have something like this, where the Scheduler is scheduling a periodic task that causes my observer to be subscribed to the observable that is my database:
_scheduler.SchedulePeriodic(_repository, TimeSpan.FromSeconds(5),
(repo) => repo.AsObservable()
.Where(item => _SomeFilter(item))
.Subscribe(item => _SomeProcessFunction(item))
);
Function names and the like omitted for brevity; repo.AsObservable() is simply a function that returns an IObservable<T> of all the items inside the repository at that point.
Now, I figure that this is the correct way of doing things, however before I came up with this solution I did come up with a different solution in which I had an Observable.Timer with the subscribed observer would subscribe to the AsObservable() return value every timer tick instead.
My question is that this seems very.. odd - why am I subscribing multiple times to the observable?
Sorry if this question is confusing, it confused me while writing it, however the schedulers are also confusing for me :P

What if you use the built in operators instead of manually scheduling tasks?
repo.AsObservable()
.Where(_SomeFilter)
// Wait 5 seconds before completing
.Concat(Observable.Empty<T>().Delay(TimeSpan.FromSeconds(5))
// Resubscribe indefinitely after source completes
.Repeat()
// Subscribe
.Subscribe(_SomeProcessFunction);

Use Rx Start, Retry, Delay, Wait for synchronous file delete retry

I need to delete a file and some other process in the application blocks it. As a workaround I decided to try several times with an interval. Is it correct approach:
Observable.Start(() => File.Delete(path)).Retry(2)
.Delay(TimeSpan.FromMilliseconds(500)).Wait();

This won't work the way you want. There are three problems:
Delay doesn't work how you think - it delays passing on the events, but the source still runs immediately.
You are issuing the Retry before the Delay
You need to use Defer to create a factory because Start will only call the embedded function once on evaluation.
Have a look at this answer for more detail on Delay and why DelaySubscription is better: Rx back off and retry.
This answer has a good implementation of a back-off retry: Write an Rx "RetryAfter" extension method
A simple fix for your code could be this, which catches the exception and rethrows it after a delay - but there's no delay if it works:
Observable.Defer(() => Observable.Start(() => File.Delete(path)))
.Catch((Exception ex) =>
Observable.Throw<Unit>(ex)
.DelaySubscription(TimeSpan.FromMilliseconds(500)))
.Retry(2)
.Wait();
Do have a look at the second link above for a fuller and better implementation though.
I kept the code above simple to make the point and isn't perfect - it always delays the exception for example.
You really want to have the DelaySubscription on the action and have it's delay time be dynamically calculated depending on the number of retries, which is what the linked implementation will do.

Strange if-statement behavior with zero value double

Would anyone care to explain to me how the value of this.oBalance.QouteBalance is evaluated to be true for being less than zero when it clearly isn't? Please see image below.
Am I missing something fundamental when it comes to comparing doubles in C#??
public double QouteBalance { get; set; }
UpdateBalance_PositionOpenned() is not being called in a loop, but is being called as part of a more complex event driven procedure that runs on the ticks of a timer (order of milliseconds)
EDIT: Pardon the code if it's messy but I couldn't edit it as this was a run-time error after quite a long run-time so was afraid wouldn't be able to recreate it. The Exception message is not correct and just a reminder for myself. The code after the exception is code I forgot to comment out before starting this particular run.
EDIT 2: I am building and running in Release Mode.
EDIT 3: Pardon my ignorance, but it would seem that I am in fact running in a multi-threaded environment since this code is being called as part of a more complex object method that gets executed on the ticks (Events) of a timer. Would it possible to ask the timer to wait until all code inside its event handler has finished before it can tick again?
EDIT 4: Since this has been established to be a multi-threading issue; I will try to give wider context to arrive at an optimized solution.
I have a Timer object, which executes the following on every tick:
Run a background worker to read data from file
When background worker finishes reading data from file, raise an
Event
In the event handler, run object code that calls the method below
(in the image) and other multiple routines, including GUI updates.
I suppose this problem can be avoided by using the timer Tick events to read the from file but changing this will break other parts of my code.

You're accessing shared variables from multiple threads. It's probably a race condition where one thread has thrown the error but by the time the debugger has caught and attached, the variable's value has changed.
You would need to look at implementing synchronizing logic like locking around the shared variables, etc.
Edit: To answer your edit:
You can't really tell the timer to not tick (well you can, but then you're starting and stopping and even after calling Stop you might still receive a few more events depending on how fast they are being dispatched). That said, you could look at Interlocked namespace and use it to set and clear and IsBusy flag. If your tick method fires and sees you're already working, it just sits out that round and waits for a future tick to handle work. I wouldn't say it's a great paradigm but it's an option.
The reason I specify using the Interlocked class versus just using a shared variable against comes down to the fact you're access from multiple threads at once. If you're not using Interlocked, you could get two ticks both checking the value and getting an answer they can proceed before they've flipped the flag to keep others out. You'd hit the same problem.
The more traditional way of synchronizing access to shared data member is with locking but you'll quickly run into problems with the tick events firing too quickly and they'll start to back up on you.
Edit 2: To answer your question about an approach to synchronizing the data with shared variables on multiple threads, it really depends on what you're doing specifically. We have a very small window into what your application is doing so I'm going to piece this together from all the comments and answers in hopes it will inform your design choice.
What follows is pseudo-code. This is based on a question you asked which suggests you don't need to do work on every tick. The tick itself isn't important, it just needs to keep coming in. Based on that premise, we can use a flagging system to check if you're busy.
...
Timer.Start(Handle_Tick)
...
public void Handle_Tick(...)
{
//Check to see if we're already busy. We don't need to "pump" the work if
//we're already processing.
if (IsBusy)
return;
try
{
IsBusy = true;
//Perform your work
}
finally
{
IsBusy = false;
}
}
In this case, IsBusy could be a volatile bool, it could be accessed with Interlocked namespace methods, it could be a locking, etc. What you choose is up to you.
If this premise is incorrect and you do in fact have to do work with every tick of the timer, this won't work for you. You're throwing away ticks that come in when you're busy. You'd need to implement a synchronized queue if you wanted to keep hold of every tick that came in. If your frequency is high, you'll have to be careful as you'll eventually overflow.

This isn't really an answer but:
UpdateBalance_PositionOpenned() is not being called in a loop, but is
being called as part of a more complex event driven procedure that
runs on the ticks of a timer (order of milliseconds)
see:
Multi-threading? – abatishchev 30 mins ago
Tight timer driven event-loop on the order of milliseconds probably has all the problems of threads, and will be almost entirely impossible to trouble-shoot with a step-through debugger. Stuff is happening way faster than you can hit 'F10'. Not to mention, you're accessing a variable from a different thread each event cycle, but there's no synchronization in sight.

Not really a full answer but too much for a comment
This is how I could code defensively
Local scope leads to less unexpected stuff
And it make code easier to debug and test
public void updateBalance(double amount, double fee, out double balance)
{
try
{
balance = amount * (1.0 + fee);
if (balance < 0.0) balance = 0.0;
}
catch (Exception Ex)
{
System.Diagnostics.Debug.WriteLine(Ex.Message);
throw Ex;
}
}
Value type is copied so even if then input variable for amount changed while the method was executing the value for amount in the method would not.
Now the out balance without locks is a different story.

Event Ordering in .NET

Got a quick question on event ordering in C#/.NET.
Let's say you have a while loop that reads a socket interface (TCP). The interface is definitely taking care of the ordering (it's TCP). Let's say your packet interface is written so that each "packet" you get in the stream, you will forward it to the next "layer" or the next object via an event callback.
So here is the pseudocode:
while (1) {
readsocket();
if (data received = complete packet)
raiseEvent(packet);
}
My questions are:
Are the events generated in order? (i.e. preserve ordering)
I am assuming #1 is correct, so that means it will block the while loop until the event finishes processing?

You never know how the event is implemented. It's possible that the events will all be executed synchronously, in order, and based on some meaningful value. It's also possible that they'll be executed synchronously in some arbitrary and inconsistent ordering. It's also possible that they won't even be executed synchronously, and that the various event handlers will be executed in new threads (or thread pool threads). It's entirely up to the implementation of the event to determine all of that.
It's rather uncommon to see different event handlers executed in parallel (and by that I mean very, very very rare), and almost all events that you come across will be backed by a single multicast delegate, meaning the order they will be fired in is the order in which they were added, but you have no way of actually knowing if that's the case (baring decompiling the code). There is no indication from the public API if that is how it is implemented.
Regardless of all of this, from a conceptual perspective, it would be best to not rely on any ordering of event handler invocations, and it's generally best to program as if the various event handlers could be run concurrently because at a conceptual level, that is what an event represents even if the implementation details are more restrictive.