I have a Windows service (c#, .net v4.5.2) that runs daily and is supposed to log errors to the Windows Event log (via System.Diagnostics) and errors plus progress information to the database (via log4net). Log4net is set to write "INFO" level.
Both logging mechanisms were working fine last time I looked, but last night the Windows service failed. I went to the d/b and event log and found that it hasn't written to either since 20th June, until it logged the automatic restart (to both logs) after it crashed last night.
Platform is a Windows 2012R2 VM in Azure. The service takes change-tracked SQL updates from a slave database, applies them to the master database, backs up the master and restores it over the slave, the log4net log is on the master. It's all been running fine, it just gave up logging.
Can anyone explain why it might have stopped logging to both log4net and the Windows event log?
The application server, domain controller and database server were all rebooted at the same time on the day log4net stopped logging. The likely cause therefore is that when the application server started up and the Windows service started, log4net was unable to connect to the database and shut down.
(Absence of Windows events probably a red herring - since the service was otherwise running happily without any errors, there just wasn't anything for it to write to Windows events.)
Related
I have a problem on my windows service that keeps writing the same logs, anyone knows the solution for the service to not keep creating same logs like this? All that aside, my service is still running smoothly but it just keeps on creating logs files. If I open the log files, the timestamp for each log is for example 8:34:42 AM
I have this Windows Service (using SqlDependency) that sends me an e-mail everytime it is started and everytime a new row is inserted on my Sales table (the database has Service Broker enabled).
I also write a log file to register when the service is started, the data is inserted to the table, if the e-mail was successfully sent and when the service is stoped.
When I started the service manually, it worked just fine.
Then I turned my PC off and on again and inserted data on my table
in the database, but nothing happened (no e-mail sent and no
register in my log file).
I restarted the service manually and it worked fine again.
Turned the PC off and on again and nothing heppened again.
I made some research and changed the initialization type to Automatic
and tried again. And nothing happened again.
I changed the initialization type to Automatic Delayed and tried
again. But it didn't work too, even 1 hour after I turned the PC on.
Now I manually restarted the service and the log file registered that
the service stoped and started, I received the email that the service
started and, after I inserted data in my database table, both log
register and email message worked just fine.
Everytime it didn't work, I checked the services manager in Windows and it was marked as "Running".
I don't know if I need to configure something different in my code or in my service properties.
Has anybody ever had this issue?
EDIT
I modified my service and included a timer, so every minute passed it writes the time in my log text file. After I restarted my PC, the log file continued to register the time every minute, but did not register the insert I made in my database table.
My guess is that the service might be losing the database connection when the PC is restarted, so the SqlDependency cannot detect the changes. Does it make any sense?
Your service depends on SQL Server, but your service is probably starting before SQL Server has been started, or finished starting. This may be causing your SqlDependency to fail.
You can mark your service as being dependent on SQL Server, which means:
Windows won't attempt to start your service until SQL Server has started and is ready, and
Starting your service will cause Windows to start SQL Server before it starts your service
This can be achieved on the command-line with the sc command (which will need to be run at an administrative command prompt). If your SQL Server instance is named MSSQLSERVER and your service is named MyService the command would look like this:
sc config MyService depend= MSSQLSERVER
Note: The space after = and before MSSQLSERVER matters. The detailed syntax can be found on learn.microsoft.com.
Once this command has run successfully, this is what you'll see when looking at the properties of your service in Control Panel > Administrative Tools > Services:
I have a web service running in IIS-10 on a Windows Server 2016 instance within a VM Hypervisor. A separate scheduled task calls functions of that web service during off peak times in order to retrieve status updates from a third party system. The scheduled task breaks the items that need to have statuses pulled into small batches and calls a function that retrieves / updates the records in parallel via Tasks and gives a return once all Tasks have completed.
Sometimes (every third time?), during this scheduled task, the app pool that the service is running on hangs. Log4Net stops logging, requests to the service do not get a response, IIS logging for the service is not updated with requests. There are no errors recorded in either my logs or in the windows event logs. When this happens, the app pool will stay in this state indefinitely. If I recycle the App Pool that the service is running on, the service will respond normally for ~30 seconds, and then the server will do a hard restart.
After the restart the event logs show the below error:
The computer has rebooted from a bugcheck. The bugcheck was: 0x00000139 (0x0000000000000003, 0xffffd60019506680, 0xffffd600195065d8, 0x0000000000000000).
The dmp file that is generated shows the same error code and identifies the file as ntoskrnl.exe.
All drivers are fully up to date. I have made sure all tasks and requests have timeouts. I have increased server resources past the point where that could be the cause. I have adjusted the batch size of items being processed.
I am out of troubleshooting ideas and would appreciate any help I can get.
I figured I would close this out in case anyone else has this very specific issue.
Digging through the dump, BHDRVX64.SYS (Symantec Antivirus) was on the stack immediately before the crash.
A 4 days later Symantec pushed an update https://support.symantec.com/en_US/article.INFO4367.html with a fix for the issue.
** If you hit a similar issue to this, start by uninstalling antivirus and seeing if the issue persists. After that, work through the list of kernel level processes returned by the 'fltmc' command in admin command prompt.
IIS 7.5 introduces the notion of auto-start providers, that allow you to get WAS to auto-load an application or assemblies when an application pool starts up.
Can a similar thing be achieved with IIS7?
Basically, we have an application that runs under WAS, and has an in-memory cache of data. When an application pool recycle occurs, my WAS deployed app won't actually be activated until the first hit for it is received. This means that the cache is cold when the first hit is received. It would be good to be able to pre-start the application as soon as the app pool is recycled.
Other options we've considered are:
Deploying the application as a Windows service so it doesn't re-cycle (this would work, but the application lifecycle management of IIS/WAS is a useful thing apart from this issue)
Writing a separate service whose job is to ping our application to warm it up.
However, the nicest way would be to get IIS7/WAS to do this for us.
In Windows 2008 you can log events that occur on the application pool, so you can log recycle events.
You can configure the event viewer to start a program when a specified message has been logged. You could call your service or load the assemblies in that program.
Would this be a feasible solution for you?
Regards,
Michel
in the advanced settings of your application pool set your application pool to generate Recycle event log entry every time it is recycled; I think the option is "Specific Time". Then you can use Windows Task Scheduler, create a script or something you want it to run that will hit your site so it can initialize. Set the trigger for the task to an even, set the Event Filter that you want to trigger the task and voila.
I have a WCF service that is running as a windows service. As soon as the WCF service starts it will try to connect to the DB and do some initialization processing.
I'm having an issue when I restart my machine (the wcf service is set to start automatically), the service would get an exception as too sql is busy. So it seems like the sql is trying to start and at the same time my service is trying to communicate with it.
For now, I did something like try/catch and I sleep for like 10 sec in my catch and I retry again until my initialization is successful.
This works pretty well. But I'm looking for a more elegant solution.
I don't want to set a dependency on sql on my windows service as the service can be installed on a different machine that the DB.
You might try configuring the "Recovery" options of your service when you deploy it to have it restart itself when it fails to load. You can do this via the SC.EXE command-line tool, WMI, or manually through the service properties (services.msc MMC).
Trying [n] times, with some timespan between retries, before ultimately failing if still unable to connect is not a bad solution if you consider monitoring and alerting.
The service actually failing to start will write warning/error events to the Windows event log, potentially triggering monitoring and alerting behavior that you might not want if its just waiting on a dependency to be available, and it will eventually recover.
HTH,
Z