Develop Reference

WCF Service hard crashing Windows 2016

I have a web service running in IIS-10 on a Windows Server 2016 instance within a VM Hypervisor. A separate scheduled task calls functions of that web service during off peak times in order to retrieve status updates from a third party system. The scheduled task breaks the items that need to have statuses pulled into small batches and calls a function that retrieves / updates the records in parallel via Tasks and gives a return once all Tasks have completed.
Sometimes (every third time?), during this scheduled task, the app pool that the service is running on hangs. Log4Net stops logging, requests to the service do not get a response, IIS logging for the service is not updated with requests. There are no errors recorded in either my logs or in the windows event logs. When this happens, the app pool will stay in this state indefinitely. If I recycle the App Pool that the service is running on, the service will respond normally for ~30 seconds, and then the server will do a hard restart.
After the restart the event logs show the below error:
The computer has rebooted from a bugcheck. The bugcheck was: 0x00000139 (0x0000000000000003, 0xffffd60019506680, 0xffffd600195065d8, 0x0000000000000000).
The dmp file that is generated shows the same error code and identifies the file as ntoskrnl.exe.
All drivers are fully up to date. I have made sure all tasks and requests have timeouts. I have increased server resources past the point where that could be the cause. I have adjusted the batch size of items being processed.
I am out of troubleshooting ideas and would appreciate any help I can get.

I figured I would close this out in case anyone else has this very specific issue.
Digging through the dump, BHDRVX64.SYS (Symantec Antivirus) was on the stack immediately before the crash.
A 4 days later Symantec pushed an update https://support.symantec.com/en_US/article.INFO4367.html with a fix for the issue.
** If you hit a similar issue to this, start by uninstalling antivirus and seeing if the issue persists. After that, work through the list of kernel level processes returned by the 'fltmc' command in admin command prompt.

C# console app using Interop.domino.dll - failing when running as Scheduled Task

I have written a C# console application that uses the Interop.domino.dll assembly to interact with domino / notes to create, update and delete documents. The application runs successfully when running it manually through a cmd prompt or through Visual Studio. However, when we try to set the application up as a scheduled task (running under a domain service account) it fails with the following error:
System.Runtime.InteropServices.COMException: Retrieving the COM class factory for component with CLSID {29131539-2EED-1069-BF5D-00DD011186B7} failed due to the following
error: 80004005 Unspecified error (Exception from HRESULT: 0x80004005 (E_FAIL)).
The error occurs as soon as we try to use an object from the Interop.domino.dll, specifically when try to create an instance of the NotesSession object as per the following:
ISession notesSession = new NotesSession();
After some investigation I found that the interop assembly requires a desktop to interact with and that if we ran the scheduled task with the service account logged into the machine that the scheduled was running on the scheduled task would run successfully. While logged in as the service account you could see that the application would bring up a svchost.exe window while the application was running and that is the reason it requires the desktop.
However, having the service account logged into the machine all the time is not an acceptable solution as it means that the account has to be logged in again each time the server restarts. There are also some security concerns around having about allowing a service account to login to machines.
So, I was wondering if anyone had any suggestions on how to get around this issue? Is there a way to suppress any UI that the assembly tries to show? Alternatively, can anyone suggest an alternate to a scheduled task which would achieve a similar result. What we need to do is:
Have the application run at a set interval (IE - once a day / once an hour)
Ideally, have it run under a domain service account (as opposed to a local system account)
Run without requiring the service account to be logged into the machine / UI elements from the Interop.domino.dll suppressed
We have thought about writing a similar application making use of the notes web services rather than the Interop assmebly and will go down that path if we can't get the console app running as a scheduled task. However, we would like to make use of what I have already written if possible.
Update 01/05/12
I have tried etting the NOTESNTSERVICE OS environment variable as per #dna-man solution. However, this did not solve the issue.
For now we have set the application up as a windows service running under the local system account and with interactivity with the desktop allowed. This is not ideal as it does not allow us to schedule it as easily and it is not running under a domain service account, but the application does work so we will go with this approach for now.
There was an answer that suggested this approach, but it seems to have been removed so I can't mark it as correct. I might leave the question open a bit to see if anyone else has any suggestions.

If your code is running on a Domino server you must make sure the OS environment variable NOTESNTSERVICE=1 is set. You can find out more abou this environment variable in the Lotus C API documentation, but it applies to the COM API as well when running as a service. I had created way back in 2003 a VB.NET application that had to run as a service, and setting this environment variable was the key. If it wasn't set, the service would stop working as soon as I logged off the machine. To make sure somebody didn't forget to set this environment variable when installing the service in the future I simply had my service manager start code set this environment variable directly on startup using the VB.NET call to SetEnvironmentVariable. It had to be set before creation of the Domino session object.

It's hard to say exactly why, but I believe it's failing to read the registry information for the Lotus interop classes. Obviously the entries are there, otherwise it would not work while running manually. So let's consider other possibilities.
I don't even know if this is possible, but os your application running as 32 bit when run manually, but 64 bit when scheduled? IBM doesn't support the Domino COM classes on Win64, and I'm pretty sure that the first symptom of that is that the registry l
A more mundane possibility is that the service account just doesn't have access to read the registry entries for the Notes/Domino install on the machine, possibly because the software was installed under a specific user account.

I ended up modifying the application slightly to set it up as a windows service running under the local system account and with interactivity with the desktop allowed. This is not ideal as it does not allow us to schedule it as easily and it is not running under a domain service account. However, this approach does allow the application to run without requiring an account to always be logged in and does allow for the interop UI elements to be shown (thus avoiding the error).
To schedule the windows service to perform the task at a set interval (which I set at once a day) I used the Timer solution proposed here.

Windows Service launching Out of Process COM server - CSIDL_APPDATA returning as different user path

I have a windows service written in C#. This service spawns a number of MTA threads. In each of those threads an out of process COM class is instantiated. I set the service to run as my current user (add the appropriate log in credentials). When the service starts it appears from Task manager that it is running as my user. If the service does a check for the local roaming user directory it returns my user directory as expected. When my COM class which is running as an out of process exe does this check, it returns a path in Windows\System32\Config\systemprofile\AppData\Roaming.
Can anyone shed some light on what is going on here, and even more specifically, how can I get the out of process server to run as the service user. I have checked in the task manager and according to task manager the exe is running as the same user (in the same session) as the service.
Not sure if the OS is a factor, but I have seen this on both Vista 32 (sp2) and Win7 64 (sp1)

You can get the server to run as a specific account using RunAs. Not very flexible but seems to do what you want here.

C# application keeps freezing on remote

I am developing a C# application (.Net 3.5, Win Forms) which is run on a server, and is accessed by Users using remote desktop. The application keeps freezing on seemingly random occasions on the remote machine (i.e. all GUI components turn to white, task manager reports the application to be not responding), but not when run locally (I'm not entirely sure about that, but failed to reproduce the freeze on my machine).
Has anyone experienced such behavior in his apps that are accessed remotely? What debugging strategy would you suggest? Do I need to consider something special when developing Win Forms applications that are accessed by remote desktop?
EDIT: some notes about the application and the freeze: The application does not recover from the freeze. Also, the freeze does not happen (or did not happen yet) during user interaction, but in between log ins to the remote machine. The application monitors a CFD solver, so it's doing things even when no one is using it.
UPDATE:
We did infact implement detailed logging, writing every function call to a file with a timestamp. Unfortunately, the results were not very conclusive. I.e. the last function call logged always returned correctly. Also, there were some background timers still running, even though the application appeared frozne (GUI completely white etc.). After some trouble we managed to have a look at a crashdump in WinDBG. On the system thread we found a call to OnUserPreferenceChanged() and further up to Invoke.WaitOne(). We can't say for sure yet, but it seems to be the issue decribed in these articles. As a quick fix, I installed a dummy handler to the event mentioned. I'll report how this works out.
UPDATE 2:
As it turns out, a Log In to a remote machine fires several OnUserPreferenceChanged() events. So it was indeed the suspected issue. The fix turned out to be not so easy though. I would have expected that an IllegalCrossReferenceException is thrown everytime a background thread tries to modify a control that was created on the system thread. This does not seem to be the case. I named my system thread and before each access to a control I asserted that the current thread name is the system thread's name. In various places this assertion failed (e.g. in a callback from a timer), but no exception was thrown. After using proper delegation at these places, the freezes stopped. The application runs nonstop for some weeks now and my users are happy again ;)

I do not think the freeze has anything to do with remote desktop. Adding logging was a good suggestion. I have a few suggestions, but without knowing the details of you application I can't get too specific.
The simplest suggestion I have is to check the memory useage and CPU usage in task manager when the freeze occurs.
If adding detailed logging is not an option, add just enough logging to know WHEN the application freezes. This could simply be a thread in the application which writes a timestamp to a file once a minute. Then you can see if there is any pattern in when it freezes, such as after a user has logged off, or when some of the data which you are monitoring changes, or at a certain time each day, or after being online for a certain amount of time.
A final and very hacky solution is to write a small watch dog application. This application's only job is to periodically check on the main application to make sure it is still responsive. How you dow this various drastically based on what your application does. If the watchdog sees the the main application has stopped, it can kill the thread for the main application and re-start it from the binaries.

if your using application streaming your server could be slowing the connection or waiting for packets that are dropping this can create this if your physical using windows remote desktop then there is your problem intense applications are not supposed to be run though remote desktop

AFAIK, there is no difference. Additionaly, I never experienced such a problem. I suggest, you try the following:
Extend your application with extended logging, to see what the users are doing when your application freezes
Check the network connection that is used to connect to the remote machine
Check the CPU usage during the freeze
If the freeze is for an extended amount of time, try to do the following:
Reproduce the freeze via remote desktop.
Go to the machine you just reproduced the freeze on and log in directly and see, whether the application is still freezed

If it is windows 10 version 1903 plus version. Then probably its the issue of windows. Please try the following settings:
Have read various blogs and concluded that the GPO has to force to use XDDM rather than WDDM. The issue is observed with windows 10 version 1903. Currently tested the workaround on machine NDT-61. It is working fine. Please practice the same for respective environment and let me know if some issue is faced.
Steps to Follow:
Open Group Policy Editor
Explore Administrative Templates
Explore Windows Components
Explore Remote Desktop Service
Click Remote Session Environment
Under Remote Session Environment Disable Use WDDM graphics display drive for Remote Desktop Connections
Note:
This policy setting lets you enable WDDM graphics display driver for Remote Desktop Connections.
If you enable or do not configure this policy setting, Remote Desktop Connections will use WDDM graphics display driver.
If you disable this policy setting, Remote Desktop Connections will NOT use WDDM graphics display driver. In this case, the Remote Desktop Connections will use XDDM graphics display driver.
For this change to take effect, you must restart Windows
Observation:
The policy settings are only available in windows 10 version 1903 and plus [ideally since when the issue started].

How to keep a process running on a remote windows server

I need to implement a background process that runs on a remote windows server 24/7. My development environment is C#/ASP.NET 3.5. The purpose of the process is to:
Send reminder e-mails to employees and customers at appropriate times (say 5:00PM on the day before a job is scheduled)
Query and save GPS coordinates of employees when they are supposed to be out on jobs so that I can later verify that their positions were where they were supposed to be.
If the process fails (which it probably will, especially when updates are added), I need for it to be restarted immediately (or within just a few minutes) as I would have very serious problems if this process failed to send a notification, log a GPS coordinate, or any of the other tasks its meant to perform.

Implement your process as a Windows service.
For a straightforward example of how
to code a Windows service this in
.Net see http://www.developer.com/net/csharp/article.php/2173801 .
To control what happens should the
service fail configure this through
the "Recovery" tab on your service
in services.msc. (see image below)
For higly critical operation you
might look into setting up a server cluster for mitigating single
server failure (see http://msdn.microsoft.com/en-us/library/ms952401.aspx ).
(source: microsoft.com)

You need a Windows Service. You can do non-visual iterative operations in windows services.

Another alternative is to create a normal application and run it on a schedule. Your application is run at certain times a day to perform its actions, depending on how often you need to log GPS coordinates and send reports. If your service doesn't need to run constantly this is usually the recommended approach, as services are supposed to be limited to always-on applications.

As well as being a service, you might want to run on a cluster, and make your service known to the cluster management software.

You can create Windows Service (server programming on Windows) or use scheduler to periodically execute a task.
Depending on the requirements for the high availability, program can be installed on a fail-over cluster where there will be other server (passive node) started and quietly waiting as a hot-backup if the first (active node) dies. This is wide topic. Start with High availablity on Wikipedia.

In my experience if you need to run something 24x7 you need to have (one or more) watchdog process to verify that your service(s) are running correctly. Just relying on the normal service framework cannot guarantee that the program is working correctly - even if it looks like it is running. The watchdog program (which also is a service) can query the service automatically e.g. posting messages checking response times, querying for statistics and so on - when it detects problems it can restart the service (or do some other fail-recovery)
The reason for having a watchdog program as opposed to just rely on user queries to detect errors is that it can be done automatically. This is the preferred method because it allows for a proactive detection.