MS AI: Duplicate event with Persistence Channel - c#

We're introducing Application Insights into out Desktop app. Since the user might be off-line when using the app, we're using a PersistenceChannel to make sure the events can be sent in a later session, and we call flush when the app is shutting down ( Dispose() of our Tracker):
public ApplicationInsightsTracker()
{
this.client = new TelemetryClient();
this.client.InstrumentationKey = InstrumentationKey;
TelemetryConfiguration.Active.TelemetryChannel = new PersistenceChannel();
TelemetryConfiguration.Active.TelemetryChannel.DeveloperMode = true;
}
~ApplicationInsightsTracker()
{
this.Dispose();
}
public override void Dispose()
{
this.client.Flush();
GC.SuppressFinalize(this);
}
public override void TrackEvent(ITrackerEvent trackerEvent)
{
try
{
this.client.TrackEvent(trackerEvent.Name, trackerEvent.Properties);
}
catch (Exception e)
{
Debug.WriteLine(string.Format("Failed to track event {0}. Exception message {1}", trackerEvent.Name, e.Message));
}
}
We're also using continuous export to send the event data from Application Insights to an Azure Blob database. We connect Power BI to the Blob database, and the other day the refresh functionality stopped working. We investigated and it turns out we were loading 2 events with the same unique ID. Looking into the blobs we found 2 consecutive blobs with the same event:
blob1.blob - Holds 1 event
{"event":...,"internal":{"data":{"id":"8709bb70-e6b1-11e5-9080-f77f0d66d988"..."data":{..."eventTime":"2016-03-10T11:15:53.9378827Z"}..."user":{..."anonId": "346033da-012d-4cc4-9841-836e5d8f8e32"..."session":{"id":"cb668d2f-9755-4afd-97c2-66cc3504349a"...
blob2.blob - Holds 3 events
{"event":...,"internal":{"data":{"id":"8709bb70-e6b1-11e5-9080-f77f0d66d988"..."data":{..."eventTime":"2016-03-10T11:15:53.9378827Z"}..."user":{..."anonId": "346033da-012d-4cc4-9841-836e5d8f8e32"..."session":{"id":"cb668d2f-9755-4afd-97c2-66cc3504349a"...
{"event":...
{"event":...
As you can see the first event on both blobs is the same. We were running tests on the PersistenceChannel having the machine connected / disconnected from the network, and somewhere along the line AI did this.
We're not entirely sure if this is a problem in how we're using it, or a flaw with the library. As you can imagine getting duplicate events through can be quite a pain (specially if you're building a model externally).
Are we doing something odd with AI, or is this a known issue?

I checked with the team that does export, and they said
the current export pipeline has opportunities for duplicate exports
And it is something they are looking into.
So it doesn't look like you are doing anything wrong, this is just a case you'll need to be aware of, and work around for now.

Exported data from AppInsights may contain dupes.
If you are exporting all your data to Power BI, then you can use Power Query's built-in duplicate removal feature.

Related

EventStore ClientAPI Subscription issue

I am facing a subscription issue with EventStore.ClientAPI. I have a projection manager that configured to subscribe all and the service started and i am getting subscriptions for events such as $metadata, $UserCreated, $statsCollected etc. but no event from the stream i created. I am pretty new here, please guide me where i am going wrong.
void InitiateProjection(IProjection projection)
{
var checkpoint = GetCurrentPosition(projection.GetType());
_eventStoreConnection.SubscribeToAllFrom(
checkpoint,
CatchUpSubscriptionSettings.Default,
EventAppeared(projection),
LiveProcessingStarted(projection));
}
Event store - product created event
What about if you specifically connect to ProductCreatedDomainEvent+d78435fc43fd-a7bf-56c01d7efa25?
As an observation, the stream name might start to give you problems (unless you changed the system proejctions to search for + instead or - already?)
If you have changed to +, could you try conencting to the $ce-ProductCreatedDomainEvent and see what you get.

How do I trigger a subscription error to test recovery on exchange web service

My application uses the EWS API with a Streaming Subscription and everything is working fine and thats a problem for me as i havn't been able to exercise my recovery code for the OnSubscriptionError event.
Here is the code i use to subscribe for streaming notifications
private void SetStreamingNotifications(List<FolderId> folder_ids)
{
streaming_subscriptions_connection = new StreamingSubscriptionConnection(exchange_service, 30);
streaming_subscriptions_connection.OnDisconnect += OnDisconnect;
streaming_subscriptions_connection.OnSubscriptionError += OnSubscriptionError;
streaming_subscriptions_connection.OnNotificationEvent += OnNotificationEvent;
foreach (var folder_id in folder_ids)
{
StreamingSubscription sub = exchange_service.SubscribeToStreamingNotifications(
new[] { folder_id },
EventType.Created,
EventType.Modified,
EventType.Deleted,
EventType.Moved,
EventType.Copied
);
streaming_subscriptions_connection.AddSubscription(sub);
}
streaming_subscriptions_connection.Open();
}
private void OnSubscriptionError(object sender, SubscriptionErrorEventArgs args)
{
/* What exceptions can i expect to find in "args.Exception" */
/* Can the streaming subscription be recovered or do i need to create a new one? */
}
So my question is how can i trigger a subscription error so i can ensure my code can recover where possible and log / alert when not possible?
EDIT
Following a comment from #kat0r i feel i should add:
I'm currently testing against Exchange 2013 and also intend to test against Exchange 2010 SP1.
I logged a call with Microsoft to find out if it was possible. The short answer is no you can't trigger the OnSubscriptionError event.
Here are the email responses from MS:
In answer to your question, I don’t believe that there is a way you can trigger the OnSubscriptionError event. The correct action to take if you do encounter this event is to attempt to recreate the subscription that encountered the error. I will see if I can find out any further information about this, but the event is generated rarely and only when an unexpected error is encountered on the Exchange server (which is why it probably isn’t possible to trigger it).
It occurred to me that the EWS Managed API has been open-sourced, and is now available on Github: https://github.com/officedev/ews-managed-api
Based on this, we can see exactly what causes the OnSubscriptionError event to be raised – and as far as I can see, this only occurs in the IssueSubscriptionFailures and IssueGeneralFailure methods, both of which can be found in StreamingSubscriptionConnection.cs. Any error that is not ServiceError.ErrorMissedNotificationEvents and is tied to a subscription will result in this event being raised (and the subscription being removed). The error is read from the Response Xml. Of course, this doesn’t really answer your question of how to trigger the event, as that involves causing Exchange to generate such an error (and I’m afraid there is no information on this). It may be possible to inject some Xml (indicating an error) into a response in a test environment – in theory, you may be able to use Fiddler to do this (as it allows you to manipulate requests/responses).
A few things you could do is
Throttling will restrict the maximum number of subscriptions you can create so if you just keep creating new subscription you should get a throttling response from the server once you exceed 20.
The other thing is if you take the SubscriptionId and use a different process to unsubscribe you other code should get a Subscription not found.
You also want to test underlying network issue eg break the connection , dns if you have dev environment see what happens when you bounce the Exchange Server etc.
Cheers
Glen

Prevent Silverlight Out-Of-Browser App from opening twice

i'm trying to prevent a Silverlight OOB App from opening twice, but i have no idea how.
I tried with creating a FileStream direct after app launch with "FileShare.None", to throw an error when the App is open twice and tries to open the file, but its ugly and doesn't work because the FileStream seems to be released after about 30 seconds..
FileStream s2 = new FileStream(path, FileMode.OpenOrCreate,FileAccess.ReadWrite, FileShare.None);
Any idea how i could approach this?
Thanks, phil
One way to achieve this is to use the local messaging channel between Silverlight applications. This scenario is mentioned in the MSDN, I will expand a little bit more here.
The LocalMessageReceiver class allow you to register to a messaging service primarily intended to communicate between different Silverlight applications.
The trick is that you can only register once with the same name in a particular scope. So as a consequence, if the first instance registers itself using your application name, any other instance doing the same afterwards will trigger an exception. Then you just have to catch that exception and deal with it, depending on what you want to do (close the instance, display a message, etc.)
Here's the code I use:
private bool CheckSingleInstance()
{
try
{
var receiver = new LocalMessageReceiver("application name", ReceiverNameScope.Global, LocalMessageReceiver.AnyDomain);
receiver.Listen();
return true;
}
catch (ListenFailedException)
{
// A listener with this name already exists
return false;
}
}
An advantage of this solution is that it works whether your instances are in-browser or out-of-browser.

Starting application from service running as SYSTEM that can interact with the user

I currently have a single application that needs to be started from a windows service that i am coding in .net 3.5. This application is currently running as the user who ran the service, in my case the SYSTEM user. If running as the SYSTEM user it does not show the application to the users desktop. Thoughts? advice?
//constructor
private Process ETCHNotify = new Process();
//StartService()
ETCHNotify.StartInfo.FileName = baseDir + "\\EtchNotify.exe";
ETCHNotify.StartInfo.UseShellExecute = false;
//BackgroundWorkerThread_DoWork()
if (!systemData.GetUserName().Equals(""))
{
// start ETCHNotify
try {
ETCHNotify.Start();
}
catch (Exception ex)
{
systemData.Run("ERR: Notify can't start: " + ex.Message);
}
}
I only execute the try/catch if the function i have written GetUserName() (which determines the username of the user running explorer.exe) is not null
again to reiterate: desired functionality is that this starts ETCHNotify in a state that allows it to interact with the currently logged in user as determined by GetUserName()
Collage of some post found around (this and this)
Note that as of Windows Vista, services are strictly forbidden from interacting directly with a user:
Important: Services cannot directly interact with a user as of Windows
Vista. Therefore, the techniques mentioned in the section titled Using
an Interactive Service should not be used in new code.
This "feature" is broken, and conventional wisdom dictates that you shouldn't have been relying on it anyway. Services are not meant to provide a UI or allow any type of direct user interaction. Microsoft has been cautioning that this feature be avoided since the early days of Windows NT because of the possible security risks.
There are some possible workarounds, however, if you absolutely must have this functionality. But I strongly urge you to consider its necessity carefully and explore alternative designs for your service.
Use WTSEnumerateSessions to find the right desktop, then CreateProcessAsUser to start the application on that desktop (you pass it the handle of the desktop as part of the STARTUPINFO structure) is correct.
However, I would strongly recommend against doing this. In some environments, such as Terminal Server hosts with many active users, determining which desktop is the 'active' one isn't easy, and may not even be possible.
A more conventional approach would be to put a shortcut to a small client app for your service in the global startup group. This app will then launch along with every user session, and can be used start other apps (if so desired) without any juggling of user credentials, sessions and/or desktops.
Ultimately in order to solve this i took the advice of #marco and the posts he mentioned. I have created the service to be entirely independent of the tray application that interacts with the user. I did however install the Tray application via registry 'start up' methods with the service. The Service installer will now install the application which interacts with the user as well... This was the safest and most complete method.
thanks for your help everyone.
I wasn't going to answer this since you already answered it, (and it's oh, what? going on 2.5 years OLD now!?) But there are ALWAYS those people who are searching for this same topic, and reading the answers...
In order to get my service to Interact with the Desktop, no matter WHAT desktop, nor, how MANY desktops, nor if the service was even running on the SAME COMPUTER as the desktop app!! None of that matters with what I got here... I won't bore you with the details, I'll just give you the meat and potatoes, and you and let me know if you want to see more...
Ok. First thing I did was create an Advertisement Service. This is a thread that the service runs, opens up a UDP socket to listen for broadcasts on the network. Then, using the same piece of code, I shared it with the client app, but it calls up Advertise.CLIENT, rather than Advertise.SERVER... The CLIENT opens the port I expect the service to be on, and broadcasts a message, "Hello... Is there anybody out there??", asking if they're there ANY servers listening, and if so, reply back to THIS IP address with your computer name, IP Address and port # where I can find the .NET remoting Services..." Then it waits a small amount of time-out time, gathers up the responses it gets, and if it's more than one, it presents the user with a dialog box and a list of services that responded... The Client then selects one, or, if only ONE responded, it will call Connect((TServerResponse) res); on that, to get connected up. At this point, the server is using Remoting Services with the WellKnownClientType, and WellKnownServerType to put itself out there...
I don't think you are too interested in my "Auto-Service locater", because a lot of people frown on UDP, even more so when your app start broadcasting on large networks. So, I'm assuming you'd be more interested in my RemotingHelper, that gets the client connected up to the server. It looks like this:
public static Object GetObject(Type type)
{
try {
if(_wellKnownTypes == null) {
InitTypeCache();
}
WellKnownClientTypeEntry entr = (WellKnownClientTypeEntry)_wellKnownTypes[type];
if(entr == null) {
throw new RemotingException("Type not found!");
}
return System.Activator.GetObject(entr.ObjectType, entr.ObjectUrl);
} catch(System.Net.Sockets.SocketException sex) {
DebugHelper.Debug.OutputDebugString("SocketException occured in RemotingHelper::GetObject(). Error: {0}.", sex.Message);
Disconnect();
if(Connect()) {
return GetObject(type);
}
}
return null;
}
private static void InitTypeCache()
{
if(m_AdvertiseServer == null) {
throw new RemotingException("AdvertisementServer cannot be null when connecting to a server.");
}
_wellKnownTypes = new Dictionary<Type, WellKnownClientTypeEntry>();
Dictionary<string, object> channelProperties = new Dictionary<string, object>();
channelProperties["port"] = 0;
channelProperties["name"] = m_AdvertiseServer.ChannelName;
Dictionary<string, object> binFormatterProperties = new Dictionary<string, object>();
binFormatterProperties["typeFilterLevel"] = "Full";
if(Environment.UserInteractive) {
BinaryServerFormatterSinkProvider binFormatterProvider = new BinaryServerFormatterSinkProvider(binFormatterProperties, null);
_serverChannel = new TcpServerChannel(channelProperties, binFormatterProvider);
// LEF: Only if we are coming form OUTSIDE the SERVICE do we want to register the channel, since the SERVICE already has this
// channel registered in this AppDomain.
ChannelServices.RegisterChannel(_serverChannel, false);
}
System.Diagnostics.Debug.Write(string.Format("Registering: {0}...\n", typeof(IPawnStatServiceStatus)));
RegisterType(typeof(IPawnStatServiceStatus),m_AdvertiseServer.RunningStatusURL.ToString());
System.Diagnostics.Debug.Write(string.Format("Registering: {0}...\n", typeof(IPawnStatService)));
RegisterType(typeof(IPawnStatService), m_AdvertiseServer.RunningServerURL.ToString());
System.Diagnostics.Debug.Write(string.Format("Registering: {0}...\n", typeof(IServiceConfiguration)));
RegisterType(typeof(IServiceConfiguration), m_AdvertiseServer.RunningConfigURL.ToString());
}
[SecurityPermission(SecurityAction.Demand, Flags=SecurityPermissionFlag.RemotingConfiguration, RemotingConfiguration=true)]
public static void RegisterType(Type type, string serviceUrl)
{
WellKnownClientTypeEntry clientType = new WellKnownClientTypeEntry(type, serviceUrl);
if(clientType != RemotingConfiguration.IsWellKnownClientType(type)) {
RemotingConfiguration.RegisterWellKnownClientType(clientType);
}
_wellKnownTypes[type] = clientType;
}
public static bool Connect()
{
// Init the Advertisement Service, and Locate any listening services out there...
m_AdvertiseServer.InitClient();
if(m_AdvertiseServer.LocateServices(iTimeout)) {
if(!Connected) {
bConnected = true;
}
} else {
bConnected = false;
}
return Connected;
}
public static void Disconnect()
{
if(_wellKnownTypes != null) {
_wellKnownTypes.Clear();
}
_wellKnownTypes = null;
if(_serverChannel != null) {
if(Environment.UserInteractive) {
// LEF: Don't unregister the channel, because we are running from the service, and we don't want to unregister the channel...
ChannelServices.UnregisterChannel(_serverChannel);
// LEF: If we are coming from the SERVICE, we do *NOT* want to unregister the channel, since it is already registered!
_serverChannel = null;
}
}
bConnected = false;
}
}
So, THAT is meat of my remoting code, and allowed me to write a client that didn't have to be aware of where the services was installed, or how many services were running on the network. This allowed me to communicate with it over the network, or on the local machine. And it wasn't a problem to have two or more people running the app, however, yours might. Now, I have some complicated callback code in mine, where I register events to go across the remoting channel, so I have to have code that checks to see if the client is even still connected before I send the notification to the client that something happened. Plus, if you are running for more than one user, you might not want to use Singleton objects. It was fine for me, because the server OWNS the objects, and they are whatever the server SAYS they are. So, my STATS object, for example, is a Singleton. No reason to create an instance of it for EVERY connection, when everyone is going to see the same data, right?
I can provide more chunks of code if necessary. This is, of course, one TINY bit of the overall picture of what makes this work... Not to mention the subscription providers, and all that.
For the sake of completeness, I'm including the code chunk to keep your service connected for the life of the process.
public override object InitializeLifetimeService()
{
ILease lease = (ILease)base.InitializeLifetimeService();
if(lease.CurrentState == LeaseState.Initial) {
lease.InitialLeaseTime = TimeSpan.FromHours(24);
lease.SponsorshipTimeout = TimeSpan.FromSeconds(30);
lease.RenewOnCallTime = TimeSpan.FromHours(1);
}
return lease;
}
#region ISponsor Members
[SecurityPermissionAttribute(SecurityAction.LinkDemand, Flags=SecurityPermissionFlag.Infrastructure)]
public TimeSpan Renewal(ILease lease)
{
return TimeSpan.FromHours(12);
}
#endregion
If you include the ISponsor interface as part of your server object, you can implement the above code.
Hope SOME of this is useful.
When you register your service, you can tell it to allow interactions with the desktop. You can read this oldie link http://www.codeproject.com/KB/install/cswindowsservicedesktop.aspx
Also, don't forget that you can have multiple users logged in at the same time.
Apparently on Windows Vista and newer interacting with the desktop has been made more difficult. Read this for a potential solution: http://www.codeproject.com/KB/cs/ServiceDesktopInteraction.aspx

What is the easiest way using .net to check if events have been logged in the eventlog?

What is the easiest way to check if events have been logged in the eventlog during a period of time?
I want to perform a series of automated test steps and then check if any errors were logged to the Application Event Log, ignoring a few sources that I'm not interested in. I can use System.Diagnostics.EventLog and then look at the Entries collection, but it doesn't seem very useable for this scenario. For instance Entries.Count can get smaller over time if the event log is removing old entries. I'd prefer some way to either query the log or monitor it for changes during a period of time. e.g.
DateTime start = DateTime.Now;
// do some stuff...
foreach(EventLogEntry entry in CleverSolution.EventLogEntriesSince(start, "Application"))
{
// Now I can do stuff with entry, or ignore if its Source is one
// that I don't care about.
// ...
}
Just to be a good Wiki citizen and strive for completion, there are other ways. I didn't suggest it earlier because it is complete overkill for something that is only going to be run in-house as part of a test suite, and you said right in the title you wanted something easy.
But if you need to see events as they occur in shipping code, read on. Believe it or not there are three different Windows APIs for this thing at this point.
NotifyChangeEventLog()
The original API for this sort of thing is called NotifyChangeEventLog() and it was supported starting in Windows 2000. Essentially you use the WIN32 event log APIs to open the event log, then you call this API with the handle you were given by the other API and an event handle. Windows will signal your event when there are new event log entries to look at.
I never used this API myself, because most of my interest was in remote event log access and this API explicitly does not support remote logs. However, the rest of the API set this belongs to does let you sequentially read remote logs if you have the right permissions.
Windows Management Instrumentation
A second way is to use the Windows Management Instrumentation API, and this does support both local and remote logs. This is a COM/DCOM based API that has existed for several years in Windows, and the .NET Framework has a nice implementation of it in the System.Management namespace. Essentially what you do is create an EventQuery that looks for the appearance of new WMI objects of type (meaning within the WMI type system) of Win32_NTLogEvent. The appearance of these will indicate new event log entries, and they will present pretty much in real time. The attributes on these objects contain all the details of the log entry. There's an article from MSDN magazine that talks about playing around with this stuff in Visual Studio.
Again, this would be total overkill for a test application, it would require far more code than your existing solution. But years ago I wrote a subsystem for a network management application that used the DCOM flavor of this API to gather the event logs off of all the servers on a network so we could alert on particular ones. It was pretty slick and darn near real time. If you implement this in C++ with DCOM, be prepared to deal with Multithreaded Apartments and a lot of hairy logic to detect if/when your connection to the remote server goes up or down.
Windows Vista Event Log
Windows Vista (and Server 2008) have a whole new API suite relating to event logging and tracing. The new event log is documented here. It looks like there is an API called EvtSubscribe that allows you to subscribe to events. I have not used this API so I can't comment on its pros and cons.
That having been said, here's an answer that actually should be pretty straightforward even for your test application and is .NET Framework specific.
You need to open the EventLog before you start your test, and subscribe an event handler to the EventLog.EntryWritten event. This is the way that .NET exposes the NotifyChangeEventLog() Win32 API.
Move your current logic from GetEventLogEntriesSince() into the event handler, but instead of adding the events to a list for return, store them in a list you can retrieve from somewhere at the end of the run. You can retrieve the contents of the log entry from the EntryWrittenEventArgs argument which is passed, via its Entry property.
The System.Diagnostics.EventLog class really is the right way to do this.
Your main objection seems to be that the log can remove old entries in some cases. But you say this is in a software testing scenario. Can't you arrange to configure your test systems such that the logs are large enough to contain all entries and the removal of old entries won't occur during your tests?
Well the solution I've come up with does use System.Diagnostics.EventLog and simply iterating over all events to filter for the ones I want. I guess this is straightforward, I just thought there would have been a more efficient interface for this. Any suggestions or improvements very welcome!
I've created a method to return event log entries since a certain time:
/// <summary>
/// Steps through each of the entries in the specified event log and returns any that were written
/// after the given point in time.
/// </summary>
/// <param name="logName">The event log to inspect, eg "Application"</param>
/// <param name="writtenSince">The point in time to return entries from</param>
/// <param name="type">The type of entry to return, or null for all entry types</param>
/// <returns>A list of all entries of interest, which may be empty if there were none in the event log.</returns>
public List<EventLogEntry> GetEventLogEntriesSince(string logName, DateTime writtenSince, EventLogEntryType type)
{
List<EventLogEntry> results = new List<EventLogEntry>();
EventLog eventLog = new System.Diagnostics.EventLog(logName);
foreach (EventLogEntry entry in eventLog.Entries)
{
if (entry.TimeWritten > writtenSince && (type==null || entry.EntryType == type))
results.Add(entry);
}
return results;
}
In my test class I store a timestamp:
private DateTime whenLastEventLogEntryWritten;
and during test setup I set the timestamp to when the last event log entry was written:
EventLog eventLog = new EventLog("Application");
whenLastEventLogEntryWritten = eventLog.Entries.Count > 0 ?
eventLog.Entries[eventLog.Entries.Count - 1] : DateTime.Now;
At the end of my test I check that there were no event log errors:
Assert.IsEmpty(GetEventLogEntriesSince("Application",
whenLastEventLogEntryWritten,
EventLogEntryType.Error),
"Application Event Log errors occurred during test execution.");
I use this Powershell to scan the eventlog for relevant entries within the last 7 days:
$d=Get-Date
$recent=[System.Management.ManagementDateTimeConverter]::ToDMTFDateTime($d.AddDays(-7))
get-wmiobject -computer HOSTNAME -class Win32_NTLogEvent `
-filter "logfile = 'Application' and (sourcename = 'SOURCENAME' or sourcename like 'OTHERSOURCENAME%') and (type = 'error' or type = 'warning') AND (TimeGenerated >='$recent')" |
sort-object #{ expression = {$_.TimeWritten} } -descending |
select SourceName, Message |
format-table #{Expression = { $_.SourceName};Width = 20;Label="SourceName"}, Message
If you use C# (tagged, but not mentioned in the question), the magic lies in the get-wmiobject query.

Categories

Resources