I am using Reactive library in my C# project to group data according to configured policies. All these data implement the following interface
public interface IPoint
{
object Value { get; }
DateTimeOffset Timestamp { get; }
}
One of the grouping policies I have to implement consists creating non-overlappable groups of a fixed time size, which looking at Reactive documentation could be achieved using Buffer(TimeSpan) function. This is exactly what I need, but instead of using a runtime calculated timestamp I need to use the ones defined in the Timestamp property of my objects.
I found this solution, which seems quite working:
public void Subscribe(Action<IEnumerable<IPoint>> callback)
{
long windowSizeTicks = TimeRange.Ticks; // TimeRange is my TimeSpan "buffer" size
dataPoints.GroupByUntil(x => x.Timestamp.Ticks / windowSizeTicks,
g => dataPoints.Where(x => x.Timestamp.Ticks / windowSizeTicks != g.Key))
.SelectMany(x => x.ToList())
.Subscribe(callback);
// dataPoints is ISubject<IPoint>
}
This solution just creates groups according to Ticks divisibility by TimeRange, which doesn't work correctly if the first item is not divisible by it.
An example to explain what I mean: considering the following points
Value: 1, Timestamp: "2021-04-26T00:00:01"
Value: 2, Timestamp: "2021-04-26T00:00:02"
Value: 3, Timestamp: "2021-04-26T00:00:03"
Value: 4, Timestamp: "2021-04-26T00:00:04"
and a "buffer size" of 2 seconds, I am expecting them to be grouped as [1, 2], [3, 4], but instead I receive [1], [2, 3], [4]. This happens because the grouping key is created considering the absolute time and not the difference from the starting of the data list.
I could save the timestamp of the first item and change the grouping function in this way, but I think (or at least I hope) there could be a better solution:
public void Subscribe(Action<IEnumerable<IPoint>> callback)
{
long windowSizeTicks = TimeRange.Ticks; // TimeRange is my TimeSpan "buffer" size
dataPoints.GroupByUntil(x => (x.Timestamp.Ticks - firstPoint.Timestamp.Ticks) / windowSizeTicks,
g => dataPoints.Where(x => (x.Timestamp.Ticks - firstPoint.Timestamp.Ticks) / windowSizeTicks != g.Key))
.SelectMany(x => x.ToList())
.Subscribe(callback);
// dataPoints is ISubject<IPoint>
}
I am a newbie at Reactive, so any helpful comment is welcome.
Thank you.
I have an observable which streams a value for each ms. , this is done every 250 ms. ( meaning 250 values in 250 ms (give or take) ).
Mock sample code :
IObservable<IEnumerable<int>> input = from _ in Observable.Interval(TimeSpan.FromMilliseconds(250))
select CreateSamples(250);
input.Subscribe(values =>
{
for (int i = 0; i < values.Count(); i++)
{
Console.WriteLine("Value : {0}", i);
}
});
Console.ReadKey();
private static IEnumerable<int> CreateSamples(int count)
{
for (int i = 0; i < 250; i++)
{
yield return i;
}
}
What i need is to create some form of process observable which process the input observable in a rate of 8 values every 33 ms
Something along the line of this :
IObservable<IEnumerable<int>> process = from _ in Observable.Interval(TimeSpan.FromMilliseconds(33))
select stream.Take(8);
I was wondering 2 things :
1) How can i write the first sample with the built in operators that reactive extensions provides ?
2) How can i create that process stream which takes values from the input stream
which with the behavior iv'e described ?
I tried using Window as a suggestion from comments below .
input.Window(TimeSpan.FromMilliseconds(33)).Take(8).Subscribe(winObservable => Debug.WriteLine(" !! "));
It seems as though i get 8 and only 8 observables of an unknown number of values
What i require is a recurrence of 8 values every 33 ms. from input observable.
What the code above did is 8 observables of IEnumrable and then stand idle.
EDIT : Thanks to James World . here's a sample .
var input = Observable.Range(1, int.MaxValue);
var timedInput = Observable.Interval(TimeSpan.FromMilliseconds(33))
.Zip(input.Buffer(8), (_, buffer) => buffer);
timedInput.SelectMany(x => x).Subscribe(Console.WriteLine);
But now it get's trickier i need for the Buffer value to calculated
i need this to be done by the actual MS passed between Intervals
when you write a TimeSpan.FromMilliseconds(33) the Interval event of the timer would actually be raised around 45 ms give or take .
Is there any way to calculate the buffer , something like PSUDO
input.TimeInterval().Buffer( s => s.Interval.Milliseconds / 4)
You won't be able to do this with any kind of accuracy with a reasonable solution because .NET timer resolution is 15ms.
If the timer was fast enough, you would have to flatten and repackage the stream with a pacer, something like:
// flatten stream
var fs = input.SelectMany(x => x);
// buffer 8 values and release every 33 milliseconds
var xs = Observable.Interval(TimeSpan.FromMilliseconds(33))
.Zip(fs.Buffer(8), (_,buffer) => buffer);
Although as I said, this will give very jittery timing. If that kind of timing resolution is important to you, go native!
I agree with James' analysis.
I'm wondering if this query gives you a better result:
IObservable<IList<int>> input =
Observable
.Generate(
0,
x => true,
x => x < 250 ? x + 1 : 0,
x => x,
x => TimeSpan.FromMilliseconds(33.0 / 8.0))
.Buffer(TimeSpan.FromMilliseconds(33.0));
I got into a Rx spree, so to speak, and this question is related to mine here and here. Nevertheless, maybe these are of help to someone as I could see them as useful variations of the same theme.
Question: How could one group a random stream of int (say, on interval [0, 10] produced on random interval) objects into groups and provide for earch group a variable number of absence of events alarms (for the lack of better definition, for futher background see linked posts). More specifically with code, how could one define multipe throttle settings per group in the following:
var idAlarmStream = idStream
.Select(i => i)
.GroupByUntil(key => key.Id, grp => grp.Throttle(Timespan.FromMilliseconds(1000))
.SelectMany(grp => grp.TakeLast(1))
.Subscribe(i => Console.WriteLine(i));
Here the subscribe function will be called if there is more than one second of absence of IDs per group. What if one would like to define three different values for the absence of events (say, one second, five seconds and ten seconds) and all cancelled when an event arrives? What I can think of are:
Split each ID in idStream into several synthetic ones and provide a bijective mapping between the real IDs and the synthetic ones. For instance in this case ID: 1 -> 100, 101, 102; ID: 2 -> 200, 201, 203 and then define a selector function in Throttle like so Func<int, Timespan>(i => /* switch(i)...*/) and then when Subscribe will be called, map the ID back. See also the linked questions for further background.
Create a nested grouping in which the IDs are grouped and then further the ID groups will be copied/replicated/forked (I don't know the proper term) into groups according to throttling values. This approach, I think, is rather complicated and I'm not sure if it would be the best one to go with. I'd sure be interested to see such a query, nevertheless.
In a more general setting, I suspect, this is a situation where there are multiple handlers per some group, albeit I haven't managed to find anything related to this.
<edit:
As a (hopefully clarifying) an example idStream pushes one ID: 1 upon which three different counters would be initiated, each of which awaiting for the next event to occur or alarming if no new ID 1 is detected in time. Counter 1 (C1) awaits for five seconds, counter 2 (C2) for seven seconds and counter 3 (C3) for ten seconds. If a new ID 1 will be received within interval [0, 5] seconds, all counters will be re-initialized with the aforementioned values and no alarm will be sent. If a new ID will be received within interval [0, 7) seconds, C1 alarms and C2 and C3 will be re-initialized. Similarly if a new ID will be received within an interval [0, 10) seconds C1 and C2 fire, but C3 gets just reinitialized.
That is, there would be multiple "absence alarms" or in general, actions taken, towards one ID given some conditions. I'm not sure what would be a good analogue... Perhaps stacking "alert lights" in a tower so that first is green, then yellow and lastly red. As absence of an ID continues longer and longer, a color after color will be lit (in this case red is the last one). Then when one ID is detected, all the lights will be turned off.
<edit 2:
Upon retrofitting James' code to the example as follows and leaving the rest as written I discovered the Subscribe will be called straight upon the first event on both of the two alarm levels.
const int MaxLevels = 2;
var idAlarmStream = idStream
.Select(i => i)
.AlarmSystem(keySelector, thresholdSelector, MaxLevels, TaskPoolScheduler.Default)
.Subscribe(i =>
{
Debug.WriteLine("Alarm on id \"{0}\" at {1}", i, DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss.fff", CultureInfo.InvariantCulture));
});
Let's see what's happening here and if MaxLevels could be provided dynamically...
<edit 3: James' code works. The problem was between the chair and the keyboard! Changing the time to something more sensible sure did help. In fact, I changed them to bigger figures, but it was .FromTicks and it escaped me for a few minutes.
This works I think - I'll try to add fuller explanation later. Each alarm level has a defined threshold (per signal group). These are expected to be of increasing duration.
The basic idea is to have the signals of all previous levels feed into the current level. The first level is a "zero" level of the signals themselves that is filtered out before the alarm stream is returned. Note that TSignal keys need to support value identity.
I'm sure there's room for simplification!
Sample unit test:
public class AlarmTests : ReactiveTest
{
[Test]
public void MultipleKeyMultipleSignalMultipleLevelTest()
{
var threshold1 = TimeSpan.FromTicks(300);
var threshold2 = TimeSpan.FromTicks(800);
var scheduler = new TestScheduler();
var signals = scheduler.CreateHotObservable(
OnNext(200, 1),
OnNext(200, 2),
OnNext(400, 1),
OnNext(420, 2),
OnNext(800, 1),
OnNext(1000, 1),
OnNext(1200, 1));
Func<int, int> keySelector = i => i;
Func<int, int, TimeSpan> thresholdSelector = (key, level) =>
{
if (level == 1) return threshold1;
if (level == 2) return threshold2;
return TimeSpan.MaxValue;
};
var results = scheduler.CreateObserver<Alarm<int>>();
signals.AlarmSystem(
keySelector,
thresholdSelector,
2,
scheduler).Subscribe(results);
scheduler.Start();
results.Messages.AssertEqual(
OnNext(700, new Alarm<int>(1, 1)),
OnNext(720, new Alarm<int>(2, 1)),
OnNext(1220, new Alarm<int>(2, 2)),
OnNext(1500, new Alarm<int>(1, 1)),
OnNext(2000, new Alarm<int>(1, 2)));
}
[Test]
public void CheckAlarmIsSuppressed()
{
var threshold1 = TimeSpan.FromTicks(300);
var threshold2 = TimeSpan.FromTicks(500);
var scheduler = new TestScheduler();
var signals = scheduler.CreateHotObservable(
OnNext(200, 1),
OnNext(400, 1),
OnNext(600, 1));
Func<int, int> keySelector = i => i;
Func<int, int, TimeSpan> thresholdSelector = (signal, level) =>
{
if (level == 1) return threshold1;
if (level == 2) return threshold2;
return TimeSpan.MaxValue;
};
var results = scheduler.CreateObserver<Alarm<int>>();
signals.AlarmSystem(
keySelector,
thresholdSelector,
2,
scheduler).Subscribe(results);
scheduler.Start();
results.Messages.AssertEqual(
OnNext(900, new Alarm<int>(1, 1)),
OnNext(1100, new Alarm<int>(1, 2)));
}
}
public static class ObservableExtensions
{
/// <summary>
/// Create an alarm system that detects signal gaps of length
/// determined by a signal key and signals alarms of increasing severity.
/// </summary>
/// <typeparam name="TSignal">Type of the signal</typeparam>
/// <typeparam name="TKey">Type of the signal key used for grouping, must implement Equals correctly</typeparam>
/// <param name="signals">Input signal stream</param>
/// <param name="keySelector">Function to select a key from a signal for grouping</param>
/// <param name="thresholdSelector">Function to select a threshold for a given signal key and alarm level.
/// Should return TimeSpan.MaxValue for levels above the highest level</param>
/// <param name="levels">Number of alarm levels</param>
/// <param name="scheduler">Scheduler use for throttling</param>
/// <returns>A stream of alarms each of which contains the signal and alarm level</returns>
public static IObservable<Alarm<TSignal>> AlarmSystem<TSignal, TKey>(
this IObservable<TSignal> signals,
Func<TSignal, TKey> keySelector,
Func<TKey, int, TimeSpan> thresholdSelector,
int levels,
IScheduler scheduler)
{
var alarmSignals = signals.Select(signal => new Alarm<TSignal>(signal, 0))
.Publish()
.RefCount();
for (int i = 0; i < levels; i++)
{
alarmSignals = alarmSignals.CreateAlarmSystemLevel(
keySelector, thresholdSelector, i + 1, scheduler);
}
return alarmSignals.Where(alarm => alarm.Level != 0);
}
private static IObservable<Alarm<TSignal>> CreateAlarmSystemLevel<TSignal, TKey>(
this IObservable<Alarm<TSignal>> alarmSignals,
Func<TSignal, TKey> keySelector,
Func<TKey, int, TimeSpan> thresholdSelector,
int level,
IScheduler scheduler)
{
return alarmSignals
.Where(alarmSignal => alarmSignal.Level == 0)
.Select(alarmSignal => alarmSignal.Signal)
.GroupByUntil(
keySelector,
grp => grp.Throttle(thresholdSelector(grp.Key, level), scheduler))
.SelectMany(grp => grp.TakeLast(1).Select(signal => new Alarm<TSignal>(signal, level)))
.Merge(alarmSignals);
}
}
public class Alarm<TSignal> : IEquatable<Alarm<TSignal>>
{
public Alarm(TSignal signal, int level)
{
Signal = signal;
Level = level;
}
public TSignal Signal { get; private set; }
public int Level { get; private set; }
private static bool Equals(Alarm<TSignal> x, Alarm<TSignal> y)
{
if (ReferenceEquals(x, null))
return false;
if (ReferenceEquals(y, null))
return false;
if (ReferenceEquals(x, y))
return true;
return x.Signal.Equals(y.Signal) && x.Level.Equals(y.Level);
}
// Equality implementation added to help with testing.
public override bool Equals(object other)
{
return Equals(this, other as Alarm<TSignal>);
}
public override string ToString()
{
return string.Format("Signal: {0} Level: {1}", Signal, Level);
}
public bool Equals(Alarm<TSignal> other)
{
return Equals(this, other);
}
public static bool operator ==(Alarm<TSignal> x, Alarm<TSignal> y)
{
return Equals(x, y);
}
public static bool operator !=(Alarm<TSignal> x, Alarm<TSignal> y)
{
return !Equals(x, y);
}
public override int GetHashCode()
{
return ((Signal.GetHashCode()*37) ^ Level.GetHashCode()*329);
}
}
I have several observables created from different events and use the sample method to create samples of equal time span. The problem is that they are created and started at different times and so I get equally spaced time series that are all shifted. I can't force the creation time, so therefore I would like to work with a reference time stamp to align all these observables.
So let's say the sample interval is 5 sec and the reference time stamp is 01-Jan-1969 00:00:00
I would get the following times series:
19-Nov-2012 08:00:00
19-Nov-2012 08:00:05
19-Nov-2012 08:00:10
....
The question is of course how to do that in RX.
The goal is to have aligned time series form events for a plot.
Here is an example:
public class Program
{
public static void Main()
{
IObservable<long> Sequence1 = Observable.Interval(TimeSpan.FromSeconds(1));
IObservable<long> Sequence2 = Observable.Interval(TimeSpan.FromSeconds(1));
//1st subscription
Sequence1.Timestamp().Buffer(TimeSpan.FromSeconds(5.0)).Subscribe(item1 =>
Console.WriteLine("Buffer 1: {0} Value: {1}", item1[item1.Count[1].
Timestamp.ToString("HH:mm:ss"), item1[item1.Count - 1].Value));
//some delay
Thread.Sleep(2000);
//2nd subscription
Sequence2.Timestamp().Buffer(TimeSpan.FromSeconds(5.0)).Subscribe(item2 =>
Console.WriteLine("Buffer 2: {0} Value: {1}", item2[item2.Count - 1].
Timestamp.ToString("HH:mm:ss"), item2[item2.Count - 1].Value));
Console.ReadLine();
}
}
Sequence 1 and 2 are different data sources and I would like to align them so that for example the buffer closes at the same time stamp.
I have a sequence of stock ticks coming in and I want to take all the data in the last hour and do some processing on it. I am trying to achieve this with reactive extensions 2.0. I read on another post to use Interval but i think that is deprecated.
Would this extension method solve your problem?
public static IObservable<T[]> RollingBuffer<T>(
this IObservable<T> #this,
TimeSpan buffering)
{
return Observable.Create<T[]>(o =>
{
var list = new LinkedList<Timestamped<T>>();
return #this.Timestamp().Subscribe(tx =>
{
list.AddLast(tx);
while (list.First.Value.Timestamp < DateTime.Now.Subtract(buffering))
{
list.RemoveFirst();
}
o.OnNext(list.Select(tx2 => tx2.Value).ToArray());
}, ex => o.OnError(ex), () => o.OnCompleted());
});
}
You are looking for the Window operators!
Here is a lengthy article I wrote on working with sequences of coincidence (overlapping windows of sequences)
http://introtorx.com/Content/v1.0.10621.0/17_SequencesOfCoincidence.html
So if you wanted to build a rolling average you could use this sort of code
var scheduler = new TestScheduler();
var notifications = new Recorded<Notification<double>>[30];
for (int i = 0; i < notifications.Length; i++)
{
notifications[i] = new Recorded<Notification<double>>(i*1000000, Notification.CreateOnNext<double>(i));
}
//Push values into an observable sequence 0.1 seconds apart with values from 0 to 30
var source = scheduler.CreateHotObservable(notifications);
source.GroupJoin(
source, //Take values from myself
_=>Observable.Return(0, scheduler), //Just the first value
_=>Observable.Timer(TimeSpan.FromSeconds(1), scheduler),//Window period, change to 1hour
(lhs, rhs)=>rhs.Sum()) //Aggregation you want to do.
.Subscribe(i=>Console.WriteLine (i));
scheduler.Start();
And we can see it output the rolling sums as it receives values.
0, 1, 3, 6, 10, 15, 21, 28...
Very likely Buffer is what you are looking for:
var hourlyBatch = ticks.Buffer(TimeSpan.FromHours(1));
Or assuming data is already Timestamped, simply using Scan:
public static IObservable<IReadOnlyList<Timestamped<T>>> SlidingWindow<T>(this IObservable<Timestamped<T>> self, TimeSpan length)
{
return self.Scan(new LinkedList<Timestamped<T>>(),
(ll, newSample) =>
{
ll.AddLast(newSample);
var oldest = newSample.Timestamp - length;
while (ll.Count > 0 && list.First.Value.Timestamp < oldest)
list.RemoveFirst();
return list;
}).Select(l => l.ToList().AsReadOnly());
}