Problem description
I am trying to debug a problem on a customer's server (Win 2012R2) where one of our .NET Web Applications is hosted in their IIS (version 8.5.9600.16384).
Recently, one of our technical supports changed a minor internal setting in the Web.config file of the application and according to them, nothing else was changed or done to the system. The application was not updated either, and had been running without any issues before.
Since that change, the AppPool in which the site is configured cannot be started anymore. According to the System Event logs, there are a few attempted starts, each resulting in a worker process crash, until the Rapid-Fail Protection of the pool stops the startup process entirely.
Also, since I'm on a customer's server, I only have limited debugging options.
Findings
Reported in the System Event log by WAS:
A process serving application pool '[our pool]' suffered a fatal communication error with the Windows Process Activation Service. The process id was '5664'. The data field contains the error number.
Application pool '[our pool]' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
A look at the corresponding Application Error log reveals the following:
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: PerfMon.dll, version: 8.0.10977.0, time stamp: 0x59cfb424
Exception code: 0xc0000409
Fault offset: 0x000f8c7b
Faulting process id: 0x1620
Faulting application start time: 0x01d423ff75bef49f
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon.dll
Report Id: b56327b3-8ff2-11e8-80d5-005056a52299
Faulting package full name:
Faulting package-relative application ID:
While googling for this problem I mainly focused on PerfMon.dll issues and the provided error code 0xc0000409, which is apparently either related to a corrupted registry entry or a Stack Buffer Overflow, but in both cases I've simply been unable to figure out what the cause could be.
I also tried the crash dump analysis described here
https://blogs.msdn.microsoft.com/parvez/2016/08/06/iis-application-pool-crash-and-debug-diag/
But in the end it seemed like it was just a round-trip back to the original problem, as this was the only error found in the dump:
The assembly instruction at PerfMon!DllGetClassObject+966fb in C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.10918.0\PerfMon.dll from Microsoft Corp. has caused an unknown exception (0xc0000409) on thread 2
Which is basically what I've gotten from the event log already.
My main question here is not only what might be the cause and a potential fix to this problem, but also how to properly find the cause/solution in the first place.
UPDATE
Using Fusion Logs, I was able to find three binding errors for System.Web.DynamicData, System.Web.Extensions and System.ServiceModel.web.
The operation failed.
Bind result: hr = 0x80004005. Unspecified error
Assembly manager loaded from: >C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable C:\Windows\SysWOW64\inetsrv\w3wp.exe
--- A detailed error log follows.
=== Pre-bind state information ===
LOG: DisplayName = System.ServiceModel.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
(Fully-specified)
LOG: Appbase = [app root path]
LOG: Initial PrivatePath = [app root bin directory]
LOG: Dynamic Base = C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\bee9e73a
LOG: Cache Base = C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\bee9e73a
LOG: AppName = b83a4ee
Calling assembly : (Unknown).
Rejecting code sharing because a dependent assembly did not match the conditional APTCA share mode
Searching for any information on this I was only able to find an old error related to MVC3/MVC4, which required a configuration change. That is unrelated to this issue, though.
This might also mean that the original error has nothing to do with the PerfMon at all and that the original error is just a weird side effect.
Just wanted to add that we had this (or a very similar) issue now on a Windows Server 2012 and resolved it by removing the Microsoft Monitoring Agent APM service as described in this rather brilliant blog post: http://kevingreeneitblog.blogspot.com/2017/03/scom-2016-agent-crashing-legacy-iis.html
(The issue described in the blog post supposedly only affects apps running on older .NET framework versions, but we experienced in with an app pool running only version 4.)
From our event log:
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x52157ba0
Faulting module name: PerfMon.dll, version: 8.0.11072.0, time stamp: 0x59c01c4c
Exception code: 0xc0000409
Fault offset: 0x000f8c7b
Faulting process id: 0x1ae8
Faulting application start time: 0x01d4eac448929ba3
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Program Files\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V8.0.11072.0\PerfMon.dll
Report Id: 8a700b62-56b7-11e9-8124-001dd8b71cba
Faulting package full name:
Related
I've gone through several posts about BadImageFormatException but I couldn't find answer to my situation.
I've a C# project that uses few 3rd party DLLs. The project's runtime is .NET Framework 4.8. I build the project using "Any CPU" configuration in VS 2022. An installer is built and the software is distributed through the installer.
On the end user's Windows 10 x64 machine, the software runs fine for a few days. After that suddenly the software stops launching. When tried to launch, Windows generates the following event logs:
Application: <application name>.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.BadImageFormatException
and
Faulting application name: <application name>.exe, version: 1.0.0.0, time stamp: 0x6215bc3a
Faulting module name: KERNELBASE.dll, version: 10.0.18362.535, time stamp: 0x5bd9df62
Exception code: 0xe0434352
Fault offset: 0x001135d2
Faulting process id: 0x610
Faulting application start time: 0x01d82c9bac0533c9
Faulting application path: C:\Program Files (x86)\<application path>
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 0c7bbce1-7a2b-47f5-a95a-083860aad054
Faulting package full name:
Faulting package-relative application ID:
Surprisingly, replacing the .exe file with the original .exe file solves the problem and makes the software starts again.
What could be the possible reason for this? How to fix it?
I've checked all of the 3rd party DLLs with file command that all of them are built as 32 bit DLL.
My wpf application uses Windows.winmd and System.Windows.Runtime dll, and I have been trying to deploy the application using Visual studio setup installer project.
When I tried to build the setup project it would silently fail and show this non-descriptive error:
ERROR: An error occurred while validating. HRESULT = '80070057
taking help from this link: Could not load file or assembly Windows.winmd' I learned that when winmd files are referenced in a wpf application than files need to be manually added to the 'Application Folder', I did so, after manually adding the .exe file it added the dlls to the Application Folder, now I can build the setup project but it still shows these following warnings:
WARNING: Unable to find dependency 'WINDOWS' (Signature='(null)' Version='255.255.255.255') of assembly 'System.Runtime.WindowsRuntime.dll'
WARNING: Unable to find dependency 'MATERIALDESIGNCOLORS' (Signature='(null)' Version='2.5.0.1205') of assembly 'MaterialDesignThemes.Wpf.DLL'
WARNING: Unable to find dependency 'WINDOWS' (Signature='(null)' Version='255.255.255.255') of assembly 'Scout_Windows.exe'
Am I adding the files the wrong?
After installing the application from the generated setup.msi file, when I launch the .exe file, it doesn't start. The event viewer shows two errors, Application error:
Faulting application name: Scout_Windows.exe, version: 1.0.0.0, time stamp: 0xaf6df283
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x015909b0
Faulting process ID: 0x2f58
Faulting application start time: 0x01d4907c33342151
Faulting application path: D:\temp\scout\Scout_Windows.exe
Faulting module path: unknown
Report ID: d0cf8414-8f12-4166-8836-ba1ec5ca8c80
Faulting package full name:
Faulting package-relative application ID:
.Net Runtime error:
Application: Scout_Windows.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.NullReferenceException
at Scout_Windows.App..ctor()
at Scout_Windows.App.Main()
here is snapshot for the contents of my Application Folder in the setup project
I have spent two days on this error! any help is appreciated.
I have a web site developed in ASP .NET MVC 5. It was working normally during last 9 month but since the web site got many active users (about 200 concurrent) it started to crash randomly in every 25~90 min. The strange part is that there is no exception in my log and the only exception that I could find is this in my event log:
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time
stamp: 0x5215df96 Faulting module name: ntdll.dll, version:
6.3.9600.16408, time stamp: 0x523d5305 Exception code: 0xc0000005 Fault offset: 0x0000000000055a0b Faulting process id: 0x2340 Faulting
application start time: 0x01d0e9f87a1fba57 Faulting application path:
c:\windows\system32\inetsrv\w3wp.exe Faulting module path:
C:\Windows\SYSTEM32\ntdll.dll Report Id:
97868357-55f2-11e5-80b8-a0d3c1fba76b Faulting package full name:
Faulting package-relative application ID:
Can any body explain me why it's happening and I can get more detail about exception and it's possible resolutions?
Update 1:
I found two hot fixes (https://support.microsoft.com/en-us/kb/2679415 and https://support.microsoft.com/en-us/kb/2640103 in Microsoft site which seems that is the case. One of them is not applicable for Windows 2012 :| the other one is not accessible to public. My emails to Microsoft site is also not answered!
Update 2:
Finally after hours and hours of research and code review I found cause of problem. In my code I was loading a DbSet from my Entity Framework db context. The records of database in that table was huge and because of this my app was crashing. And because I also had app pre initialization configured in IIS after crash again it started to load that DbSet again and here was a loop of crashing and crashing!
I have created an installer project in WPF. when I run the application on my computer, it works fine. I set a minimum requirement for the .net framework 4.5. But if I run the application on another machine (with newly installed windows 7), the application crashes.
Below is event viewer log:
Installer exception
Faulting application name: MyInstaller.exe, version: 1.0.0.0, time stamp: 0x55572605
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17514, time stamp: 0x4ce7bafa
Exception code: 0xe0434352
Fault offset: 0x0000b727
Faulting process id: 0x398
Faulting application start time: 0x01d08fca117f3e62
Faulting application path: C:\Users\self\Downloads\MyInstaller.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 517dc0dd-fbbd-11e4-a332-b083fe4d72a6
Application: MyInstaller.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.IO.FileNotFoundException
Stack:
at MyInstaller.App.Main(System.String[])
I tried to create "Window event log" inside Main() try-catch method, but the application was not able to reach to that point.
When I send myinstaller.config.exe file with project exe file in the new installed machine, then it shows error. That the application does not meet requirement.
In my machine myinstaller.config.exe is not requirement, even after I uninstall framework 4.5 and 4.0.
I am developing a web application based on "Silverlight 5".
I found a blocking bug related to the component "MultiScaleImage" that happens after a unpredictable (sometimes it takes hours, sometimes minutes) number of zooms of the image.
When the issue occurs the following messages are shown:
Windows 7 Event Log: Faulting application name: IEXPLORE.EXE, version: 11.0.9600.17126, time stamp: 0x53882e30
Faulting module name: agcore.dll, version: 5.1.30214.0, time stamp: 0x52fdb3c4
Exception code: 0xc0000005
Fault offset: 0x003a8d0d
Faulting process id: 0x18dc
Faulting application start time: 0x01cf9cf438322755
Faulting application path: C:\Program Files (x86)\Internet Explorer\IEXPLORE.EXE
Faulting module path: C:\Program Files (x86)\Microsoft Silverlight\5.1.30214.0\agcore.dll
Report Id: a9de107e-08fe-11e4-b5ce-c86000a4b136)
Visual Studio Debugger: Unhandled exception at 0x5CCD8D0D (agcore.dll) in iexplore.exe: 0xC0000005: Access violation reading location 0xFFE2E6F2.
Internet Explorer 11: Internet Explorer has stopped working.
The test has been done with Win7+SP1 with IE11 and IE8.
This bug is preventing me from releasing the application and deploying it to the customer servers.
I can't believe there's no way to solve it nor a valid workaround.