If you add expiration's to entities you are adding into Redis e.g. in ServiceStack.Redis:
redisClient.Set(elementKey, "some cached value", DateTime.Now.AddMinutes(2));
how can you then subscribe to the element's expiration. The desired outcome would be something ala:
redisClient.Subscribe(elementKey, "expire", DoSomethingBasedOnKey)
Actually you can subscribe to expired keys events, but like Matias said it may take some time until Redis will publish the event.
Redis has Keyspace notifications, you can read about it here,
Keyspace notifications allows clients to subscribe to Pub/Sub channels in order to receive events affecting the Redis data set in some way.
Type of events
Keyspace notifications are implemented sending two distinct type of events for every operation affecting the Redis data space. For instance a DEL operation targeting the key named mykey in database 0 will trigger the delivering of two messages, exactly equivalent to the following two PUBLISH commands:
PUBLISH keyspace#0:mykey del
PUBLISH keyevent#0:del mykey
So what you need is to subscribe to the channel that will publish a message on expired command of keyevent(work also when ttl is reached), it prefix will be like so:
"keyevent#0:expired"
Timing accuracy wasn't matter at my case so I've implemented it like so using the ServiceStack C# Redis client:
string EXPIRED_KEYS_CHANNEL = "__keyevent#0__:expired";
using (IRedisClient client = redisClient.GetClient())
{
using (var cacheSubscription = client.CreateSubscription())
{
cacheSubscription.OnMessage += (ch, expiredKey) =>
{
FireOnKeyExpired(expiredKey);
};
cacheSubscription.SubscribeToChannels(EXPIRED_KEYS_CHANNEL);
}
}
Update:
Make sure to configure redis.conf to allow key events on expired keys:
notify-keyspace-events Ex
Or on the fly like so(configuration may be lost when instance is restarted)
config set notify-keyspace-events Ex
Maybe you don't know that expire messages won't get exactly published when keys get expired:
Timing of expired events
Keys with a time to live associated are
expired by Redis in two ways:
When the key is accessed by a command and is found to be expired.
Via a background system that looks for expired keys in background, incrementally, in order to be able to also
collect keys that are never accessed.
I would suggest you that your best bet is using some task scheduler like the built-in Windows Task Scheduler or Quartz.NET to schedule a task to publish a message to some custom Redis PubSub channel exactly when the key must get expired.
Related
Introduction
Hello all, we're currently working on a microservice platform that uses Azure EventHubs and events to sent data in between the services.
Let's just name these services: CustomerService, OrderService and MobileBFF.
The CustomerService mainly sends updates (with events) which will then be stored by the OrderService and MobileBFF to be able to respond to queries without having to call the CustomerService for this data.
All these 3 services + our developers on the DEV environment make use of the same ConsumerGroup to connect to these event hubs.
We currently make use of only 1 partition but plan to expand to multiple later. (You can see our code is already made to be able to read from multiple partitions)
Exception
Every now and then we're running into an exception though (if it starts it usually keeps throwing this error for an hour or something). For now we've only seen this error on DEV/TEST environments though.
The exception:
Azure.Messaging.EventHubs.EventHubsException(ConsumerDisconnected): At least one receiver for the endpoint is created with epoch of '0', and so non-epoch receiver is not allowed. Either reconnect with a higher epoch, or make sure all epoch receivers are closed or disconnected.
All consumers of the EventHub, store their SequenceNumber in their own Database. This allows us to have each consumer consume events separately and also store the last processed SequenceNumber in it's own SQL database. When the service (re)starts, it loads the SequenceNumber from the db and then requests events from here onwards untill no more events can be found. It then sleeps for 100ms and then retries. Here's the (somewhat simplified) code:
var consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName;
string[] allPartitions = null;
await using (var consumer = new EventHubConsumerClient(consumerGroup, _inboxOptions.EventHubConnectionString, _inboxOptions.EventHubName))
{
allPartitions = await consumer.GetPartitionIdsAsync(stoppingToken);
}
var allTasks = new List<Task>();
foreach (var partitionId in allPartitions)
{
//This is required if you reuse variables inside a Task.Run();
var partitionIdInternal = partitionId;
allTasks.Add(Task.Run(async () =>
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await using (var consumer = new EventHubConsumerClient(consumerGroup, _inboxOptions.EventHubConnectionString, _inboxOptions.EventHubName))
{
EventPosition startingPosition;
using (var testScope = _serviceProvider.CreateScope())
{
var messageProcessor = testScope.ServiceProvider.GetService<EventHubInboxManager<T, EH>>();
//Obtains starting position from the database or sets to "Earliest" or "Latest" based on configuration
startingPosition = await messageProcessor.GetStartingPosition(_inboxOptions.InboxIdentifier, partitionIdInternal);
}
while (!stoppingToken.IsCancellationRequested)
{
bool processedSomething = false;
await foreach (PartitionEvent partitionEvent in consumer.ReadEventsFromPartitionAsync(partitionIdInternal, startingPosition, stoppingToken))
{
processedSomething = true;
startingPosition = await messageProcessor.Handle(partitionEvent);
}
if (processedSomething == false)
{
await Task.Delay(100, stoppingToken);
}
}
}
}
catch (Exception ex)
{
//Log error / delay / retry
}
}
}
}
The exception is thrown on the following line:
await using (var consumer = new EventHubConsumerClient(consumerGroup, _inboxOptions.EventHubConnectionString, _inboxOptions.EventHubName))
More investigation
The code described above is running in the MicroServices (which are hosted as AppServices in Azure)
Next to that we're also running 1 Azure Function that also reads events from the EventHub. (Probably uses the same consumer group).
According to the documentation here: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-features#consumer-groups it should be possible to have 5 consumers per consumer group. It seems to be suggested to only have one, but it's not clear to us what could happen if we don't follow this guidance.
We did do some tests with manually spawning multiple instances of our service that reads events and when there were more then 5 this resulted in a different error which stated quite clearly that there could only be 5 consumers per partition per consumer group (or something similar).
Furthermore it seems like (we're not 100% sure) that this issue started happening when we rewrote the code (above) to be able to spawn one thread per partition. (Even though we only have 1 partition in the EventHub). Edit: we did some more log-digging and also found a few exception before merging in the code to spawn one thread per partition.
That exception indicates that there is another consumer configured to use the same consumer group and asserting exclusive access over the partition. Unless you're explicitly setting the OwnerLevel property in your client options, the likely candidate is that there is at least one EventProcessorClient running.
To remediate, you can:
Stop any event processors running against the same Event Hub and Consumer Group combination, and ensure that no other consumers are explicitly setting the OwnerLevel.
Run these consumers in a dedicated consumer group; this will allow them to co-exist with the exclusive consumer(s) and/or event processors.
Explicitly set the OwnerLevel to 1 or greater for these consumers; that will assert ownership and force any other consumers in the same consumer group to disconnect.
(note: depending on what the other consumer is, you may need to test different values here. The event processor types use 0, so anything above that will take precedence.)
To add to the Jesse's answer, I think the exception message is part of
the old SDK.
If you look into the docs, there 3 types of receiving modes defined there:
Epoch
Epoch is a unique identifier (epoch value) that the service uses, to enforce partition/lease ownership.
The epoch feature provides users the ability to ensure that there is only one receiver on a consumer group at any point in time...
Non-epoch:
... There are some scenarios in stream processing where users would like to create multiple receivers on a single consumer group. To support such scenarios, we do have ability to create a receiver without epoch and in this case we allow upto 5 concurrent receivers on the consumer group.
Mixed:
... If there is a receiver already created with epoch e1 and is actively receiving events and a new receiver is created with no epoch, the creation of new receiver will fail. Epoch receivers always take precedence in the system.
We currently have a NServiceBus 5 system, which contains two recurring Sagas. Since they act as dispatcher to periodically pull multiple sorts of data from an external system, we're using the Timeouts to trigger this: We created a generic and empty class called ExecuteTask, which is used by the Saga to handle the timeout.
public class ScheduleSaga1 : Saga<SchedulerSagaData>,
IAmStartedByMessages<StartScheduleSaga1>,
IHandleMessages<StopSchedulingSaga>,
IHandleTimeouts<ExecuteTask>
And the other Saga is almost identically defined:
public class ScheduleSaga2: Saga<SchedulerSagaData>,
IAmStartedByMessages<StartScheduleSaga2>,
IHandleMessages<StopSchedulingSaga>,
IHandleTimeouts<ExecuteTask>
The timeout is handled equally in both Sagas:
public void Handle(StartScheduleSaga1 message)
{
if (_schedulingService.IsDisabled())
{
_logger.Info($"Task '{message.TaskName}' is disabled!");
}
else
{
Debugger.DoDebug($"Scheduling '{message.TaskName}' started!");
Data.TaskName = message.TaskName;
// Check to avoid that if the saga is already started, don't initiate any more tasks
// as those timeout messages will arrive when the specified time is up.
if (!Data.IsTaskAlreadyScheduled)
{
// Setup a timeout for the specified interval for the task to be executed.
Data.IsTaskAlreadyScheduled = true;
// Send the first Message Immediately!
SendMessage();
// Set the timeout
var timeout = _schedulingService.GetTimeout();
RequestTimeout<ExecuteTask>(timeout);
}
}
}
public void Timeout(ExecuteTask state)
{
if (_schedulingService.IsDisabled())
{
_logger.Info($"Task '{Data.TaskName}' is disabled!");
}
else
{
SendMessage();
// Action that gets executed when the specified time is up
var timeout = _schedulingService.GetTimeout();
Debugger.DoDebug($"Request timeout for Task '{Data.TaskName}' set to {timeout}!");
RequestTimeout<ExecuteTask>(timeout);
}
}
private void SendMessage()
{
// Send the Message to the bus so that the handler can handle it
Bus.Send(EndpointConfig.EndpointName, Activator.CreateInstance(typeof(PullData1Request)));
}
Now the problem: Since both Sagas are requesting Timeouts for ExecuteTask, it gets dispatched to both Sagas!
Even worse, it seems like the stateful Data in the Sagas gets messed up, since both Sagas are sending both message.
Therefore, it seems like the Timeouts are getting sent to all the Saga Instances which are requesting it.
But looking at the example https://docs.particular.net/samples/saga/simple/ there is no special logic regarding multiple Saga instances and their state.
Is my assumption correct? If this is the case, what are the best practices to have multiple Sagas requesting and receiving Timeouts?
The only reason I can think of when this is happening is that they share the same identifier to uniquely identify the saga instance.
Both ScheduleSaga1 and ScheduleSaga2 are using the same SchedulerSagaData for storing state. NServiceBus sees an incoming message and tries to retrieve the state, based on the unique identifier in the incoming message. If both StartScheduleSaga1 and StartScheduleSaga2 come in with identifier 1 for example, NServiceBus will search for saga state in the table SchedulerSagaData with unique identifier 1.
Both ScheduleSaga1 and ScheduleSaga2 will then share the same row!!!
Timeouts are based on SagaId in the TimeoutEntity table. Because both sagas share the same SagaId, it's logical they are both executed once the timeout arrives.
At the minimum you should not reuse the identifier to schedule tasks. It's probably better to not share the same class for storing saga state. Also easier to debug.
My application uses the EWS API with a Streaming Subscription and everything is working fine and thats a problem for me as i havn't been able to exercise my recovery code for the OnSubscriptionError event.
Here is the code i use to subscribe for streaming notifications
private void SetStreamingNotifications(List<FolderId> folder_ids)
{
streaming_subscriptions_connection = new StreamingSubscriptionConnection(exchange_service, 30);
streaming_subscriptions_connection.OnDisconnect += OnDisconnect;
streaming_subscriptions_connection.OnSubscriptionError += OnSubscriptionError;
streaming_subscriptions_connection.OnNotificationEvent += OnNotificationEvent;
foreach (var folder_id in folder_ids)
{
StreamingSubscription sub = exchange_service.SubscribeToStreamingNotifications(
new[] { folder_id },
EventType.Created,
EventType.Modified,
EventType.Deleted,
EventType.Moved,
EventType.Copied
);
streaming_subscriptions_connection.AddSubscription(sub);
}
streaming_subscriptions_connection.Open();
}
private void OnSubscriptionError(object sender, SubscriptionErrorEventArgs args)
{
/* What exceptions can i expect to find in "args.Exception" */
/* Can the streaming subscription be recovered or do i need to create a new one? */
}
So my question is how can i trigger a subscription error so i can ensure my code can recover where possible and log / alert when not possible?
EDIT
Following a comment from #kat0r i feel i should add:
I'm currently testing against Exchange 2013 and also intend to test against Exchange 2010 SP1.
I logged a call with Microsoft to find out if it was possible. The short answer is no you can't trigger the OnSubscriptionError event.
Here are the email responses from MS:
In answer to your question, I don’t believe that there is a way you can trigger the OnSubscriptionError event. The correct action to take if you do encounter this event is to attempt to recreate the subscription that encountered the error. I will see if I can find out any further information about this, but the event is generated rarely and only when an unexpected error is encountered on the Exchange server (which is why it probably isn’t possible to trigger it).
It occurred to me that the EWS Managed API has been open-sourced, and is now available on Github: https://github.com/officedev/ews-managed-api
Based on this, we can see exactly what causes the OnSubscriptionError event to be raised – and as far as I can see, this only occurs in the IssueSubscriptionFailures and IssueGeneralFailure methods, both of which can be found in StreamingSubscriptionConnection.cs. Any error that is not ServiceError.ErrorMissedNotificationEvents and is tied to a subscription will result in this event being raised (and the subscription being removed). The error is read from the Response Xml. Of course, this doesn’t really answer your question of how to trigger the event, as that involves causing Exchange to generate such an error (and I’m afraid there is no information on this). It may be possible to inject some Xml (indicating an error) into a response in a test environment – in theory, you may be able to use Fiddler to do this (as it allows you to manipulate requests/responses).
A few things you could do is
Throttling will restrict the maximum number of subscriptions you can create so if you just keep creating new subscription you should get a throttling response from the server once you exceed 20.
The other thing is if you take the SubscriptionId and use a different process to unsubscribe you other code should get a Subscription not found.
You also want to test underlying network issue eg break the connection , dns if you have dev environment see what happens when you bounce the Exchange Server etc.
Cheers
Glen
Having set up a ReferenceDataRequest I send it along to an EventQueue
Service refdata = _session.GetService("//blp/refdata");
Request request = refdata.CreateRequest("ReferenceDataRequest");
// append the appropriate symbol and field data to the request
EventQueue eventQueue = new EventQueue();
Guid guid = Guid.NewGuid();
CorrelationID id = new CorrelationID(guid);
_session.SendRequest(request, eventQueue, id);
long _eventWaitTimeout = 60000;
myEvent = eventQueue.NextEvent(_eventWaitTimeout);
Normally I can grab the message from the queue, but I'm hitting the situation now that if I'm making a number of requests in the same run of the app (normally around the tenth), I see a TIMEOUT EventType
if (myEvent.Type == Event.EventType.TIMEOUT)
throw new Exception("Timed Out - need to rethink this strategy");
else
msg = myEvent.GetMessages().First();
These are being made on the same thread, but I'm assuming that there's something somewhere along the line that I'm consuming and not releasing.
Anyone have any clues or advice?
There aren't many references on SO to BLP's API, but hopefully we can start to rectify that situation.
I just wanted to share something, thanks to the code you included in your initial post.
If you make a request for historical intraday data for a long duration (which results in many events generated by Bloomberg API), do not use the pattern specified in the API documentation, as it may end up making your application very slow to retrieve all events.
Basically, do not call NextEvent() on a Session object! Use a dedicated EventQueue instead.
Instead of doing this:
var cID = new CorrelationID(1);
session.SendRequest(request, cID);
do {
Event eventObj = session.NextEvent();
...
}
Do this:
var cID = new CorrelationID(1);
var eventQueue = new EventQueue();
session.SendRequest(request, eventQueue, cID);
do {
Event eventObj = eventQueue.NextEvent();
...
}
This can result in some performance improvement, though the API is known to not be particularly deterministic...
I didn't really ever get around to solving this question, but we did find a workaround.
Based on a small, apparently throwaway, comment in the Server API documentation, we opted to create a second session. One session is responsible for static requests, the other for real-time. e.g.
_marketDataSession.OpenService("//blp/mktdata");
_staticSession.OpenService("//blp/refdata");
The means one session operates in subscription mode, the other more synchronously - I think it was this duality which was at the root of our problems.
Since making that change, we've not had any problems.
My reading of the docs agrees that you need separate sessions for the "//blp/mktdata" and "//blp/refdata" services.
A client appeared to have a similar problem. I solved it by making hundreds of sessions rather than passing in hundreds of requests in one session. Bloomberg may not be to happy with this BFI (brute force and ignorance) approach as we are sending the field requests for each session but it works.
Nice to see another person on stackoverflow enjoying the pain of bloomberg API :-)
I'm ashamed to say I use the following pattern (I suspect copied from the example code). It seems to work reasonably robustly, but probably ignores some important messages. But I don't get your time-out problem. It's Java, but all the languages work basically the same.
cid = session.sendRequest(request, null);
while (true) {
Event event = session.nextEvent();
MessageIterator msgIter = event.messageIterator();
while (msgIter.hasNext()) {
Message msg = msgIter.next();
if (msg.correlationID() == cid) {
processMessage(msg, fieldStrings, result);
}
}
if (event.eventType() == Event.EventType.RESPONSE) {
break;
}
}
This may work because it consumes all messages off each event.
It sounds like you are making too many requests at once. BB will only process a certain number of requests per connection at any given time. Note that opening more and more connections will not help because there are limits per subscription as well. If you make a large number of time consuming requests simultaneously, some may timeout. Also, you should process the request completely(until you receive RESPONSE message), or cancel them. A partial request that is outstanding is wasting a slot. Since splitting into two sessions, seems to have helped you, it sounds like you are also making a lot of subscription requests at the same time. Are you using subscriptions as a way to take snapshots? That is subscribe to an instrument, get initial values, and de-subscribe. If so, you should try to find a different design. This is not the way the subscriptions are intended to be used. An outstanding subscription request also uses a request slot. That is why it is best to batch as many subscriptions as possible in a single subscription list instead of making many individual requests. Hope this helps with your use of the api.
By the way, I can't tell from your sample code, but while you are blocked on messages from the event queue, are you also reading from the main event queue while(in a seperate event queue)? You must process all the messages out of the queue, especially if you have outstanding subscriptions. Responses can queue up really fast. If you are not processing messages, the session may hit some queue limits which may be why you are getting timeouts. Also, if you don't read messages, you may be marked a slow consumer and not receive more data until you start consuming the pending messages. The api is async. Event queues are just a way to block on specific requests without having to process all messages from the main queue in a context where blocking is ok, and it would otherwise be be difficult to interrupt the logic flow to process parts asynchronously.
I am attempting to build an application that can monitor multiple remote machines through WMI. As a C# developer, I have chosen to utilize the System.Management namespace.
For performance and scalability reasons, I would much prefer to use an event-driven method of gathering information than a poll-based one. As such, I have been investigating the ManagementEventWatcher class.
For simple monitoring tasks, this class seems to be exactly what I want. I create the object, give it ManagementScope, EventQuery, and EventWatcherOptions parameters, subscribe to the EventArrived event, and call the Start method (simplified example below).
using SM = System.Management;
...
SM.ManagementEventWatcher _watcher;
SM.ConnectionOptions conxOptions;
SM.ManagementScope scope;
SM.WqlEventQuery eventQuery;
SM.EventWatcherOptions eventOptions;
SM.EventArrivedEventHandler handler;
string path = #"\\machine\root\cimv2";
conxOptions = new SM.ConnectionOptions ();
conxOptions.Username = user;
conxOptions.Password = password;
scope = new SM.ManagementScope (path, conxOptions);
scope.Connect ();
eventQuery = new SM.WqlEventQuery ("SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE TargetInstance ISA 'Win32_Process'");
eventOptions = new SM.EventWatcherOptions ();
eventOptions.Context.Add ("QueryName", "Process Query");
_watcher = new SM.ManagementEventWatcher (scope, eventQuery, eventOptions);
handler = new SM.EventArrivedEventHandler (HandleWMIEvent);
_watcher.EventArrived += handler;
_watcher.Start ();
Console.WriteLine ("Press Any Key To Continue");
Console.ReadKey ();
_watcher.Stop ();
_watcher.EventArrived -= handler;
The problem I am running into is that it is difficult to detect when the connection to the remote machine has been broken through various means (machine restart, downed router, unplugged network cable, etc.).
The ManagementEventWatcher class does not appear to provide any means of determining that the connection has been lost, as the Stopped event will not fire when this occurs. The ManagementScope object attached to the ManagementEventWatcher still shows IsConnected as true, despite the broken link.
Does anyone have any ideas on how to check the connection status?
The only thing I can think to do at this point is to use the ManagementScope object to periodically perform a WMI query against the machine and make sure that still works, though that can only check the local->remote connection and not the corresponding remote->local connection. I suppose I could look up another WMI query I could use to verify the connection (assuming the query works), but that seems like more work than I should have to do.
There are two kinds of event consumers in WMI - temporary and permanent. What you might be looking for is a permanent event subscription. Here is a brief blurb about that on MSDN
A permanent consumer is a COM object that can receive a WMI event at all times. A permanent event consumer uses a set of persistent objects and filters to capture a WMI event. Like a temporary event consumer, you set up a series of WMI objects and filters that capture a WMI event. When an event occurs that matches a filter, WMI loads the permanent event consumer and notifies it about the event. Because a permanent consumer is implemented in the WMI repository and is an executable file that is registered in WMI, the permanent event consumer operates and receives events after it is created and even after a reboot of the operating system as long as WMI is running. For more information, see Receiving Events at All Times.
This MSDN article should be enough to get you going http://msdn.microsoft.com/en-us/library/aa393014(VS.85).aspx.
However, in my situation in dealing with this problem, we chose to poll for the data as opposed to creating a permanent consumer. Another option is to monitor for certain events (such as a reboot) and then re-register your temporary event consumer.
Check out this post here. It covers how to detect when a removable disk is inserted using C#. SHould be inline with your WMI code that you supplied.
Subscribe to the NetworkAvailabilityChange event, this should let you know about the status of your current connection through the NetworkAvailabilityEventArgs.IsAvailable property. With a little extra work the NetworkAddressChange event will let you know about machines that move about, change addresses and etc on your network . . . The System.Net.NetworkInformation has good information . . . I' assuming you don't mind using something other than WMI to monitor this.
As far as I can tell, when something like that happens you should receive an exception of type ManagementException that contains wbemErrCallCancelled WMI error code (0x80041032).