I'm trying to implement a timeout when there no events in a period if time.
Scenario:
I have an object that raises an event every time a message is received.
I would like to react when there are no messages (OnReceived events) received in a period of time (lets say, 20 secs)
This what I have so far
var observable = Observable.FromEventPattern<BasicDeliverEventHandler>(
handler => _innerConsumer.Received += OnReceived,
handler => _innerConsumer.Received -= OnReceived);
var timeout = observable.Timeout(TimeSpan.FromSeconds(20));
using (timeout.Subscribe(_ => { },
exception =>
Tracer.Warning("Eventing Consumer timeout : {0}", exception.Message)))
{ }
I'm creating an observable from a EventPattern. Then, using the timeout. What I don't understand is how to get the exception from the Timeout. I want to react when that´s happen.
I don't think the Subcribe method is the correct way, but that's I get from the docs.
I'm open to suggestions or other alternatives if this is not the correct one.
Thanks in advance
Timeout is problematic because it terminates the sequence. Throttle is what you want - but you also need to insert a start element in case you get no events at all.
I convert the events to Unit.Default - this is useful when you don't care what happened, just that something happened - and use StartWith to seed the throttle:
var timeout = observable.Select(_ => Unit.Default)
.StartWith(Unit.Default)
.Throttle(TimeSpan.FromSeconds(20);
var subs = timeout.Subscribe(_ => Console.WriteLine("Timeout!"));
Out of interest, I also have a similar solution to this for detecting disconnected clients - this time providing a single timeout notification for multiple sources: http://www.zerobugbuild.com/?p=230
Let's take a look at the code you have.
var observable =
Observable.FromEventPattern<BasicDeliverEventHandler>(
handler => _innerConsumer.Received += OnReceived,
handler => _innerConsumer.Received -= OnReceived
);
var timeout = observable.Timeout(TimeSpan.FromSeconds(20));
using (timeout.Subscribe(
_ => { },
exception =>
Tracer.Warning("Eventing Consumer timeout : {0}", exception.Message)))
{
}
We can re-write the subscription logic like so:
var subscription = timeout.Subscribe(
_ => { }
exception =>
Tracer.Warning("Eventing Consumer timeout : {0}", exception.Message)
);
subscription.Dispose(); // This is bad
Since your subscription is being disposed of immediately, your observer isn't receiving an of the notifications you're expecting.
By removing subscription.Dispose(), or the using statement, your observer should receive a TimeoutException 20 seconds after subscribing. However, because Exceptions also cancel subscriptions, you will only ever receive this Exception once.
Furthermore, the Timeout operator starts a timeout at the time of subscription, and does not cancel the timeout unless the subscription is cancelled, or the source observer completes.
You might want to try using a different operator, such as Throttle.
observable.Throttle(TimeSpan.FromSeconds(20))
.Subscribe(x =>
Console.WriteLine("it has been 20 seconds since we received the last notification.")
)
Related
I need to listen UDP socket and after 10 seconds or 100 items in buffer some logic should be invoke. Generally it works ok, but i don't know how to proper stop listening socket.
var ip = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 1234);
var socket = new UdpClient(ip);
var cancellationTokenSource = new CancellationTokenSource();
var observable =
Observable
.FromAsync(socket.ReceiveAsync)
.DoWhile(() => !cancellationTokenSource.IsCancellationRequested)
.Buffer(TimeSpan.FromSeconds(10), 100);
var subscribtion = observable.Subscribe(o =>
{
//logic
});
//simulate close method from another thread
Task.Factory.StartNew(() =>
{
Task.Delay(TimeSpan.FromSeconds(12)).Wait();
cancellationTokenSource.Cancel();
socket.Close();
subscribtion.Dispose();
});
When i simulate closing socket, there exists a situation when exists some data in buffer that can't be processed - is any way to avoid this behavior?
When i send some messages from another process with 500ms delay, it will be works like example below:
20 messages will income
Some logic will be invoke - subscriber logic
4 messages will income
Simulate close method will be invoke
When "close method" will be invoke i need immediately process all data in buffer and close application without waiting even for buffer timeout. Buffer delay time is defined by user, so i don't want wait for invoke subscriber logic, because it can be a quite long time.
Welcome to StackOverflow!
Existing overloads of the buffer method don't support time, count and a gate. But there is an overload which triggers buffer closing when a sequence produces a value. So we will simply create a sequence by merging observables of all the conditions for buffer closing.
Have a look at this demo.
onTime will produce a value after the period specified.
onCount will produce its first value after x items have passed.
onClose will immediately produce a value on subscribe - but we won't connect to it until we decide to.
var producer = Observable.Interval(TimeSpan.FromSeconds(0.2));
var source = producer.Publish().RefCount();
var onClose = Observable.Return(0L).Publish();
var onTime = Observable.Timer(TimeSpan.FromSeconds(2));
var onCount = source.Skip(10);
var bufferClose = Observable.Merge(onClose, onTime, onCount);
var subscription =
source
.Buffer(() => bufferClose)
.Subscribe(list => Console.WriteLine(string.Join(",", list)));
Console.WriteLine("Waiting for close");
Console.ReadLine();
onClose.Connect(); //signal
subscription.Dispose();
Console.WriteLine("Closed");
Console.ReadLine();
This will produce an output based on count or time, until return is pressed, when it immediately closes with what's available in the buffer.
Consider the following:
[Fact]
public void foo()
{
var result = new Subject<bool>();
var startCount = 0;
var completionCount = 0;
var obs = Observable
.Defer(() =>
{
++startCount;
return result.FirstAsync();
})
.Do(_ => ++completionCount)
.Publish()
.RefCount();
// pretend there are lots of subscribers at once
var s1 = obs.Subscribe();
var s2 = obs.Subscribe();
var s3 = obs.Subscribe();
// even so, we only expect to be started once
Assert.Equal(1, startCount);
Assert.Equal(0, completionCount);
// and we won't complete until the result ticks through
result.OnNext(true);
Assert.Equal(1, startCount);
Assert.Equal(1, completionCount);
s1.Dispose();
s2.Dispose();
s3.Dispose();
// now try exactly the same thing again
s1 = obs.Subscribe();
s2 = obs.Subscribe();
s3 = obs.Subscribe();
// startCount is 4 here instead of the expected 2!
Assert.Equal(2, startCount);
Assert.Equal(1, completionCount);
result.OnNext(true);
Assert.Equal(2, startCount);
Assert.Equal(2, completionCount);
s1.Dispose();
s2.Dispose();
s3.Dispose();
}
My understanding of Publish + RefCount is that a connection to the source is maintained as long as there is at least one subscriber. Once the last subscriber disconnects, any future subscriber will re-initiate the connection to the source.
As you can see in my test, everything works perfectly the first time through. But the second time, the deferred observable inside the pipeline is executed once for every new subscriber.
I can see via the debugger that for the first group of subscribers, obs._count (which counts subscribers) increases for each call to Subscribe. But for the second group of subscribers, it remains zero.
Why is this happening and what can I do to rectify my pipeline?
The answer from #user631090 is close, but incorrect, so I thought I'd answer myself.
It's because Publish will immediately complete new subscribers if the stream it published has itself completed. You can kind of see that in the diagram here:
But it would have been nice if the diagram included a subscriber after the underlying stream completes.
To add to the confusion, Defer is still called for new subscribers. But its return value is simply ignored by Publish because of the initial stream completing.
I'm as yet unable to come up with a way to implement my intended use case. I thought perhaps using Multicast rather than Publish, creating a new subject as necessary. But I haven't been able to achieve that yet. And it seems rather painful for what I would think is a common use case.
It's because the underlying observable result has already completed. So each new subscriber is just getting the OnCompleted callback.
If ObservableDefer was creating a new sequence each time or one that didn't complete you would see the desired behavior.
e.g.
return result.FirstAsync().Concat(Observable.Never<bool>());
You will need to remove the Assert.Equal(1, completionCount);
I have a project where I need to send a status message every 10 seconds unless there's been an update in the meantime. Meaning, every time there would be an update, the timer would reset.
var res = Observable
.Interval(TimeSpan.FromSeconds(10))
.Where(_ => condition);
res.Subscribe(_ => Console.WriteLine("Status sent."));
Now I know that the "Where" will only be applied when the timer ends, so it doesn't help. But, I'm wondering if there's a way to reset the Interval; or to use a Timer() with a repeat.
This is pretty easy to implement using standard Rx operators.
What isn't clear from your question is exactly what an "update" is. I'm going to assume that you have some sort of observable that fires for every update or that you can create a subject that you'll call .OnNext(...) when there is an update. Without observable updates it is hard to know when to reset the timer.
So here's the code:
var update = new Subject<bool>();
var res =
update
.Select(x => Observable.Interval(TimeSpan.FromSeconds(10.0)))
.Switch();
res
.Subscribe(_ => Console.WriteLine("Status sent."));
update.OnNext(true);
The res query now waits until it gets a value from update and then it selects a new Observable.Interval. This means that after the Select the type is an IObservable<IObservable<long>>, so the .Switch() is required to turn it in to a IObservable<long>. .Switch() does this by only passing out values from the latest observed observable and disposing of any previous observables. In other words, for each update a new timer is started and the previous timer is cancelled. This means that if you have updates occurring more frequently than 10 seconds then the timer will never fire.
Now, if the res observable is an update in its own right, then you can do this:
res
.Subscribe(_ =>
{
update.OnNext(true);
Console.WriteLine("Status sent.");
});
That's fine - it still works, but for each timer firing res will create a new timer. It will mean that anything relying on your update observable/subject will still function correctly.
I keep this little helper method with me:
public static IObservable<long> CreateAutoResetInterval<TSource>(IObservable<TSource> resetter, TimeSpan timeSpan, bool immediate = false)
{
return resetter.Select(_ => immediate ? Observable.Interval(timeSpan).StartWith(0) : Observable.Interval(timeSpan)).Switch();
}
It's basically the same mechanism as Enigmativity's answer
I think you could also get away with using Throttle here. The purpose of Throttle is not to let elements though if another element is received within the given timespan. So in your case if an update message is received within 10 secs then don't send a status. See unit test below which uses 200 ticks as the throttle period.
[TestMethod]
public void Publish_Status_If_Nothing_Receieved()
{
//Arrange
var scheduler = new TestScheduler();
var statusObserver = scheduler.CreateObserver<Unit>();
var updateStream = scheduler.CreateColdObservable(OnNext(100, 1), OnNext(200, 2), OnNext(600, 3),
OnNext(700, 4));
var waitTime = TimeSpan.FromTicks(200);
//Act
updateStream.Throttle(waitTime, scheduler)
.Select(_ => Unit.Default)
.Subscribe(statusObserver);
//Verify no status received
scheduler.AdvanceTo(100);
Assert.AreEqual(0, statusObserver.Messages.Count);
//Verify no status received
scheduler.AdvanceTo(200);
Assert.AreEqual(0, statusObserver.Messages.Count);
//Assert status received
scheduler.AdvanceTo(400);
statusObserver.Messages.AssertEqual(OnNext(400, Unit.Default));
//Verify no more status received
scheduler.AdvanceTo(700);
statusObserver.Messages.AssertEqual(OnNext(400, Unit.Default));
}
I am trying to queue up items from an Azure Service Bus so I can process them in bulk. I am aware that the Azure Service Bus has a ReceiveBatch() but it seems problematic for the following reasons:
I can only get a max of 256 messages at a time and even this then can be random based on message size.
Even if I peek to see how many messages are waiting I don't know how many RequestBatch calls to make because I don't know how many messages each call will give me back. Since messages will keep coming in I can't just continue to make requests until it's empty since it will never be empty.
I decided to just use the message listener which is cheaper than doing wasted peeks and will give me more control.
Basically I am trying to let a set number of messages build up and
then process them at once. I use a timer to force a delay but I need
to be able to queue my items as they come in.
Based on my timer requirement it seemed like the blocking collection was not a good option so I am trying to use ConcurrentBag.
var batchingQueue = new ConcurrentBag<BrokeredMessage>();
myQueueClient.OnMessage((m) =>
{
Console.WriteLine("Queueing message");
batchingQueue.Add(m);
});
while (true)
{
var sw = WaitableStopwatch.StartNew();
BrokeredMessage msg;
while (batchingQueue.TryTake(out msg)) // <== Object is already disposed
{
...do this until I have a thousand ready to be written to DB in batch
Console.WriteLine("Completing message");
msg.Complete(); // <== ERRORS HERE
}
sw.Wait(MINIMUM_DELAY);
}
However as soon as I access the message outside of the OnMessage
pipeline it shows the BrokeredMessage as already being disposed.
I am thinking this must be some automatic behavior of OnMessage and I don't see any way to do anything with the message other than process it right away which I don't want to do.
This is incredibly easy to do with BlockingCollection.
var batchingQueue = new BlockingCollection<BrokeredMessage>();
myQueueClient.OnMessage((m) =>
{
Console.WriteLine("Queueing message");
batchingQueue.Add(m);
});
And your consumer thread:
foreach (var msg in batchingQueue.GetConsumingEnumerable())
{
Console.WriteLine("Completing message");
msg.Complete();
}
GetConsumingEnumerable returns an iterator that consumes items in the queue until the IsCompleted property is set and the queue is empty. If the queue is empty but IsCompleted is False, it does a non-busy wait for the next item.
To cancel the consumer thread (i.e. shut down the program), you stop adding things to the queue and have the main thread call batchingQueue.CompleteAdding. The consumer will empty the queue, see that the IsCompleted property is True, and exit.
Using BlockingCollection here is better than ConcurrentBag or ConcurrentQueue, because the BlockingCollection interface is easier to work with. In particular, the use of GetConsumingEnumerable relieves you from having to worry about checking the count or doing busy waits (polling loops). It just works.
Also note that ConcurrentBag has some rather strange removal behavior. In particular, the order in which items are removed differs depending on which thread removes the item. The thread that created the bag removes items in a different order than other threads. See Using the ConcurrentBag Collection for the details.
You haven't said why you want to batch items on input. Unless there's an overriding performance reason to do so, it doesn't seem like a particularly good idea to complicate your code with that batching logic.
If you want to do batch writes to the database, then I would suggest using a simple List<T> to buffer the items. If you have to process the items before they're written to the database, then use the technique I showed above to process them. Then, rather writing directly to the database, add the item to a list. When the list gets 1,000 items, or a given amount of time elapses, allocate a new list and start a task to write the old list to the database. Like this:
// at class scope
// Flush every 5 minutes.
private readonly TimeSpan FlushDelay = TimeSpan.FromMinutes(5);
private const int MaxBufferItems = 1000;
// Create a timer for the buffer flush.
System.Threading.Timer _flushTimer = new System.Threading.Timer(TimedFlush, FlushDelay.TotalMilliseconds, Timeout.Infinite);
// A lock for the list. Unless you're getting hundreds of thousands
// of items per second, this will not be a performance problem.
object _listLock = new Object();
List<BrokeredMessage> _recordBuffer = new List<BrokeredMessage>();
Then, in your consumer:
foreach (var msg in batchingQueue.GetConsumingEnumerable())
{
// process the message
Console.WriteLine("Completing message");
msg.Complete();
lock (_listLock)
{
_recordBuffer.Add(msg);
if (_recordBuffer.Count >= MaxBufferItems)
{
// Stop the timer
_flushTimer.Change(Timeout.Infinite, Timeout.Infinite);
// Save the old list and allocate a new one
var myList = _recordBuffer;
_recordBuffer = new List<BrokeredMessage>();
// Start a task to write to the database
Task.Factory.StartNew(() => FlushBuffer(myList));
// Restart the timer
_flushTimer.Change(FlushDelay.TotalMilliseconds, Timeout.Infinite);
}
}
}
private void TimedFlush()
{
bool lockTaken = false;
List<BrokeredMessage> myList = null;
try
{
if (Monitor.TryEnter(_listLock, 0, out lockTaken))
{
// Save the old list and allocate a new one
myList = _recordBuffer;
_recordBuffer = new List<BrokeredMessage>();
}
}
finally
{
if (lockTaken)
{
Monitor.Exit(_listLock);
}
}
if (myList != null)
{
FlushBuffer(myList);
}
// Restart the timer
_flushTimer.Change(FlushDelay.TotalMilliseconds, Timeout.Infinite);
}
The idea here is that you get the old list out of the way, allocate a new list so that processing can continue, and then write the old list's items to the database. The lock is there to prevent the timer and the record counter from stepping on each other. Without the lock, things would likely appear to work fine for a while, and then you'd get weird crashes at unpredictable times.
I like this design because it eliminates polling by the consumer. The only thing I don't like is that the consumer has to be aware of the timer (i.e. it has to stop and then restart the timer). With a little more thought, I could eliminate that requirement. But it works well the way it's written.
Switching to OnMessageAsync solved the problem for me
_queueClient.OnMessageAsync(async receivedMessage =>
I reached out to Microsoft about the BrokeredMessage being disposed issue on MSDN, this is the response:
Very basic rule and I am not sure if this is documented. The received message needs to be processed in the callback function's life time. In your case, messages will be disposed when async callback completes, this is why your complete attempts are failing with ObjectDisposedException in another thread.
I don't really see how queuing messages for further processing helps on the throughput. This will add more burden to client for sure. Try processing the message in the async callback, that should be performant enough.
In my case that means I can't use ServiceBus in the way I wanted to, and I have to re-think how I wanted things to work. Bugger.
I had the same issue when started to work with Azure Service Bus service.
I have found that method OnMessage always dispose BrokedMessage object. The approach proposed by Jim Mischel didn't help me (but it was very interesting to read - thanks!).
After some investigation I have found that the whole approach is wrong. Let me explain the right way to do what you want.
Use BrokedMessage.Complete() method only inside OnMessage method handler.
If you need to process message outside of this method that you should use method QueueClient.Complete(Guid lockToken). "LockToken" is property of BrokeredMessage object.
Example:
var messageOptions = new OnMessageOptions {
AutoComplete = false,
AutoRenewTimeout = TimeSpan.FromMinutes( 5 ),
MaxConcurrentCalls = 1
};
var buffer = new Dictionary<string, Guid>();
// get message from queue
myQueueClient.OnMessage(
m => buffer.Add(key: m.GetBody<string>(), value: m.LockToken),
messageOptions // this option says to ServiceBus to "froze" message in he queue until we process it
);
foreach(var item in buffer){
try {
Console.WriteLine($"Process item: {item.Key}");
myQueueClient.Complete(item.Value);// you can also use method CompleteBatch(...) to improve performance
}
catch{
// "unfroze" message in ServiceBus. Message would be delivered to other listener
myQueueClient.Defer(item.Value);
}
}
My solution was to get the message SequenceNumber then defer the message and add the SequenceNumber the BlockingCollection. Once the BlockingCollection picks up a new item it can receive the deferred message by the SequenceNumber and mark the message as complete. If for some reason the BlockingCollection doesn't process the SequenceNumber it will remain in the queue as deferred so it can be picked up later when the process is restarted. This protects against loosing messages if the process abnormally terminates while there's still items in the BlockingCollection.
BlockingCollection<long> queueSequenceNumbers = new BlockingCollection<long>();
//This finds any deferred/unfinished messages on startup.
BrokeredMessage existingMessage = client.Peek();
while (existingMessage != null)
{
if (existingMessage.State == MessageState.Deferred)
{
queueSequenceNumbers.Add(existingMessage.SequenceNumber);
}
existingMessage = client.Peek();
}
//setup the message handler
Action<BrokeredMessage> processMessage = new Action<BrokeredMessage>((message) =>
{
try
{
//skip deferred messages if they are already in the queueSequenceNumbers collection.
if (message.State != MessageState.Deferred || (message.State == MessageState.Deferred && !queueSequenceNumbers.Any(x => x == message.SequenceNumber)))
{
message.Defer();
queueSequenceNumbers.Add(message.SequenceNumber);
}
}
catch (Exception ex)
{
// Indicates a problem, unlock message in queue
message.Abandon();
}
});
// Callback to handle newly received messages
client.OnMessage(processMessage, new OnMessageOptions() { AutoComplete = false, MaxConcurrentCalls = 1 });
//start the blocking loop to process messages as they are added to the collection
foreach (var queueSequenceNumber in queueSequenceNumbers.GetConsumingEnumerable())
{
var message = client.Receive(queueSequenceNumber);
//mark the message as complete so it's removed from the queue
message.Complete();
//do something with the message
}
I needed to alternate between two states with each state having a different interval time.
The best way I could think of doing this was to use Reactive Extensions' Observable.Generate
which is pretty awsome.
From what I read on msdn and other sites, Observable.Finally() should fire if the
observable "terminates gracefully or exceptionally". I was testing the following code
(in LINQPad) to see how it works, but I can not get .Finall() to fire at all.
var ia = TimeSpan.FromSeconds(1);
var ib = TimeSpan.FromSeconds(.2);
var start = DateTime.UtcNow;
var ct = new CancellationTokenSource();
var o = Observable.Generate(
true,
// s => !ct.IsCancellationRequested,
s => (DateTime.UtcNow-start) < TimeSpan.FromSeconds(3) && !ct.IsCancellationRequested,
s => !s,
s => s ? "on" : "off",
s => s? ib : ia)
// .TakeUntil(start+TimeSpan.FromSeconds(3))
.Concat(Observable.Return("end"));
o.Subscribe( s=> s.Dump(), ct.Token);
var t = o.ToTask(ct.Token);
t.ContinueWith(x => x.Dump("done"));
o.Finally(() => "finallY".Dump()); // never gets called?
Thread.Sleep(10000);
ct.Cancel();
If I make Thread.Sleep 10s, the observable sequence finishes and the Task.ContinueWith fires,
but not .Finally().
If I make Thread.Sleep 2s, the observable sequence is canceled and the Task.ContinueWith again fires,
but not .Finally().
Why not?
Look at the return type of the Finally method; should give you a hint. Just like the Concat method returns a new IObservable with the new sequence concatenated to it, but doesn't change the original, the Finally method returns a new IObservable that has that final action, but you're subscribing to the original IObservable. Put the following line in front of your Subscribe call and it'll work.
o = o.Finally(() => "finallY".Dump());
I agree it's an odd API choice though; I'd think of Finally as being more akin to Subscribe than to Concat. You're subscribing to the finally "event"; it's odd that the API forces you to create a completely new IObservable and then subscribe to that just to get the Finally thing to happen. Plus it allows a potential error (made evident if we use the function in your question) that if you subscribe twice to that new IObservable, your Finally function will execute twice. So you have to make sure that one of your subscriptions is on the "finallied" IObservable and the others are all on the original. Just seems unusual.
I guess the way to think about it is that Finally isn't meant to modify the observable, but rather to modify the subscription itself. i.e., they don't expect you typically to make openly-accessible named observables that have Finally things (var o = Observable.[...].Finally(...);) rather it's meant to go inline with the subscription call itself (var subscription = o.Finally(...).Subscribe(...);)