We are getting error on server and our service is automatically stopped in the server.
Randomly application is crash in approx 1 hour with below Error as -
Faulting application name: Chubb.Studio.Event.Processor.exe, version:
0.0.0.0, time stamp: 0x5c0ab1b7 Faulting module name: KERNELBASE.dll, version: 6.3.9600.19425, time stamp: 0x5d26b6e9 Exception code:
0xc0000005 Fault offset: 0x0000000000001556 Faulting process id:
0x115c Faulting application start time: 0x01d5a35fd202f96d Faulting
application path:
E:\WindowsService\DevInt\Chubb.Studio.EventProcessor\Chubb.Studio.Event.Processor.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll Report Id:
762c15d4-0f5b-11ea-8120-005056a27597 Faulting package full name:
Faulting package-relative application ID:
Our Code is look like as -
protected override void OnStarted()
{
//IntializeEventsExecution();
Task task = Task.Factory.StartNew(() => IntializeEventsExecution());
base.OnStarted();
}
public void IntializeEventsExecution()
{
StartEvents();
}
public void StartEvents()
{
var eventList = GetEventTopics();
Parallel.ForEach(eventList,
new ParallelOptions { MaxDegreeOfParallelism = eventList.Count },
(item, state, index) =>
{
StartProcessingEvent(eventList[(int)index]);
});
}
/// <summary>
///
/// </summary>
/// <param name="index"></param>
public void StartProcessingEvent(EventTopic topic)
{
try
{
Task task = Task.Factory.StartNew(() => ExecuteProcessingEvent(topic));
task.Wait();
}
catch (Exception)
{
}
finally
{
new _processingDelegate(StartProcessingEvent).Invoke(topic);
}
}
As Klaus says in his comment, a STATUS_ACCESS-VIOLATION exception is caused by a process reading or writing memory that it doesn't own. Given this is C#, the most likely reason is either an incorrect use of P/Invoke or using unsafe code.
The best approach to debugging something vague like this is to isolate the issue by removing P/Invoke calls one by one until the exception doesn't happen. It's hard to be more precise because the exception may be triggered a long way from the cause (memory or stack corruption).
This SO answer gives a good list of the likely causes of an access violation in managed code.
Access violations in managed apps typically happen for one of these
reasons:
You P/Invoke a native API passing in a handle to a managed object and the native API uses that handle. If you get a collection and
compaction while the native API is running, the managed object may
move and the pointer becomes invalid.
You P/Invoke something with a buffer that is too small or smaller than the size you pass in and the API overruns a read or write
A pointer (IntPtr, etc) you pass to a P/Invoke call is invalid (-1 or 0) and the native isn't checking it before use
You P/Invoke a native call and the native code runs out of memory (usually virtual) and isn't checking for failed allocations and
reads/writes to an invalid address
You use a GCHandle that is not initialized or that somehow is pointing to an already finalized and collected object (so it's not
pointing to an object, it's pointing to an address where an object
used to be)
Your app uses a handle to something that got invalidated by a sleep/wake. This is more esoteric but certainly happens. For example,
if you're running an application off of a storage card, the entire app
isn't loaded into RAM. Pieces in use are demand-paged in for
execution. This is all well and good. Now if you power the device off,
the drivers all shut down. When you power back up, many devices simply
re-mount the storage devices. When your app needs to demand-page in
more program, it's no longer where it was and it dies. Similar
behavior can happen with databases on mounted stores. If you have an
open handle to the database, after a sleep/wake cycle the connection
handle may no longer be valid.
Related
With .NET 7's NativeAOT compilation. We can now load a C# dll as regular Win32 module.
HMODULE module = LoadLibraryW("AOT.dll");
auto hello = GetProcAddress(module, "Hello");
hello();
This works fine and prints some stuff in console.
However, when unloading the dll. It simply doesn't work. No matter how many times I call FreeLibrary("AOT.dll"), GetModuleHandle("AOT.dll") still returns the handle to the module, implying that it did not unload successfully.
My "wild guess" was that the runtime has some background threads still running (GC?), so I enumerated all threads and use NtQueryInformationThread to retrive the start address of each thread then call GetModuleHandleEx with GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS to get the module where the thread started, the result were as follows.
Before:
THREAD ID = 7052
base priority = 8
delta priority = 0
Start address: 00007FF69D751613
Module: 00007FF69D740000 => CppRun.exe
THREAD ID = 3248
base priority = 8
delta priority = 0
Start address: 00007FFEF1F42B20
Module: 00007FFEF1EF0000 => ntdll.dll
THREAD ID = 7160
base priority = 8
delta priority = 0
Start address: 00007FFEF1F42B20
Module: 00007FFEF1EF0000 => ntdll.dll
After:
THREAD ID = 7052
base priority = 8
delta priority = 0
Start address: 00007FF69D751613
Module: 00007FF69D740000 => CppRun.exe
THREAD ID = 3248
base priority = 8
delta priority = 0
Start address: 00007FFEF1F42B20
Module: 00007FFEF1EF0000 => ntdll.dll
THREAD ID = 7160
base priority = 8
delta priority = 0
Start address: 00007FFEF1F42B20
Module: 00007FFEF1EF0000 => ntdll.dll
THREAD ID = 5944
base priority = 8
delta priority = 0
Start address: 00007FFEF1F42B20
Module: 00007FFEF1EF0000 => ntdll.dll
THREAD ID = 17444
base priority = 10
delta priority = 0
Start address: 00007FFE206DBEF0
Module: 00007FFE206D0000 => AOT.dll
"CppRun.exe" is my testing application.
As you can see, two additional threads were spawned. One from ntdll (5944), and one from my AOT compiled dll (17444).
I don't know what the leftover thread in "AOT.dll" was for (maybe GC?), but I force-terminated it successfully (definitely unhealthy, I know).
However, when I tried to open the thread in ntdll (5944), it throws an exception
An invalid thread, handle %p, is specified for this operation. Possibly, a threadpool worker thread was specified
Given that, I assume .NET starts a threadpool worker during initilization? How can I stop that pool and unload the dll?
Or, is there a better way for unloading a NativeAOT compiled dll?
Update: I've hooked the CreateThreadPool function, but the runtime doesn't call it. Still trying to figure out what spawned that thread.
Edit:
NativeAOT(aka CoreRT) compiled dll was unloadable at first, but Microsoft later blocked the functionality due to memory leak and crash on process exit. See this PR for more details.
This answer simply restores the functionality using detour hook and does not deal with the memory leak nor the crash. Use it at your own risk.
I was able to prevent the access violation crash by manually freeing the FLS(fiber-local storage) created by .NET. Here is a simple demo.
Original answer below:
Turns out that thread is used by Windows 10 for parallel library loading(TppWorkerThread) and isn't the problem.
I ended up inspecting the winapi call with this handy tool, and found that .NET is calling GetModuleHandleEx with the GET_MODULE_HANDLE_EX_FLAG_PIN flag, thus preventing the module from unloading.
So I hooked GetModuleHandleEx to intercept calls and shift out the flag.
Voila! Now I can unload the NativeAOT compiled dll.
I know this approach is quite hacky, but at least it works.
If anyone happen to have a better solution, please let me know.
UWP app with RichEditBox has memory and other type issues on release configuration with code optimization enabled. On debug or relase with non optimized code it runs ok. Following code is inside a method running on the thread pool (await Task.Run(() => MyMethod(richEditTextDocument));
// Getting text for first time works
richEditTextDocument.GetText(Windows.UI.Text.TextGetOptions.None, out string rtbOriginalText);
foreach (Match v in wordMatches)
{
try
{
// In release mode with optimized code,
//at very first iteration, line below throws
//"Insufficient memory to continue execution of the program"
Windows.UI.Text.ITextRange selectedTextNew = richEditTextDocument.GetRange(v.Index, v.Index + v.Length);
}
catch
{
continue; //insufficient memory
}
}
// In release with optimized code, calling GetText for second time throws
//(Exception from HRESULT: 0x8000FFFF)
// at System.Runtime.InteropServices.McgMarshal.ThrowOnExternalCallFailed(Int32, RuntimeTypeHandle) + 0x21
// at __Interop.ComCallHelpers.Call(__ComObject, RuntimeTypeHandle, Int32, TextGetOptions, Void *) + 0xc2
// at __Interop.ForwardComStubs.Stub_67[TThis](__ComObject, TextGetOptions, String &, Int32) + 0x44
// at Windows.UI.Text.RichEditTextDocument.GetText(TextGetOptions, String &) + 0x23
// Thinking that it can be fixed setting selection to 0 before GetText, but...
richEditTextDocument.Selection.StartPosition = 0; //, this line throws insufficient memory
richEditTextDocument.Selection.EndPosition = 0;
// HRESULT: 0x8000FFFF (if execution reachs here, deleting two previous lines)
richEditTextDocument.GetText(Windows.UI.Text.TextGetOptions.None, out string rtbOriginalTextAnother);
Submission to store was rejected two times because of other minor errors that went fixed, but at third time it passed the test and was published, whithout noticing this error, that is part of the main function of the app and lets it "unusable" (as they (Microsoft) said the other times). Submitting a non optimized code building (but with NET native toolchain) complains about missing DEBUG dlls. I noticed the error but, as disabling code optimization when debugging the release "fixed" it (as is explained at https://devblogs.microsoft.com/devops/debugging-net-native-windows-universal-apps/ linked by official Microsoft docs at https://learn.microsoft.com/en-us/windows/msix/package/packaging-uwp-apps), i forgot that it only was being "ignored". So, first time publishing and got an unusable app.
App uses nuget packages NewtonSoft.Json, Win2D.uwp, Microsoft.NETCore.UniversalWindowsPlatform and a "normal" reference to Microsoft.Advertising.Xaml (also app is not showing ads in production, ErrorOcurred gives NoAdAvailable)
Thanks
I am trying to throw together a proof of concept project, just to see how good Microsoft's Cognitive Services Speech Transcription is.
I have followed all the examples on their site, but have so far been unsuccessful. Initially I was unable to get it to run at all under one of my existing code bases as x86, it was throwing the error:
An attempt was made to load a program with an incorrect format
Then I created a brand new .net framework x64 console app. And it would start, then crash internally using version 1.4.0 as well as a few other versions I tried and put this error into my event log:
Faulting application name: dotnet.exe, version: 2.1.27415.1, time
stamp: 0x5c672873 Faulting module name:
Microsoft.CognitiveServices.Speech.core.dll, version: 1.3.1.28, time
stamp: 0x5c764ab1 Exception code: 0xc0000094 Fault offset:
0x000000000007567c Faulting process id: 0x6200 Faulting application
start time: 0x01d4f1518c240c4b Faulting application path: C:\Program
Files\dotnet\dotnet.exe Faulting module path:
C:\Users\username.nuget\packages\microsoft.cognitiveservices.speech\1.3.1\runtimes\win-x64\native\Microsoft.CognitiveServices.Speech.core.dll
Finally I found version 1.1.0 which would actually start, (version 1.0.0 would not even allow the app to compile). Now I am running into the issue that the SessionStarted and SessionStopped events are called instantly, but no transcription ever takes place, and using Fiddler it looks like no calls are being made outside of my machine.
Unless Cognitive Services is really buggy, then there must be something simple I am missing. Can anyone point it out?
My goal is to transcribe a 5 minute or less audio file on my local network. Here is the code I am attempting.
static void Main(string[] args)
{
Console.WriteLine("Hello World!");
var file = #"U:\path\file.wav";
ContinuousRecognitionAsync(file).Wait();
Console.WriteLine("End!");
}
public static async Task ContinuousRecognitionAsync(string audiopath)
{
// subscription key and service region. Replace with your own subscription key
// and service region (e.g., "westus").
var config = SpeechConfig.FromSubscription("<my free test key>", "westus");
var audio = Microsoft.CognitiveServices.Speech.Audio.AudioConfig.FromWavFileInput(audiopath);
// Creates a continuos speech recognizer using WAV input.
using (var recognizer = new SpeechRecognizer(config, audio))
{
//Subscribes to events.
recognizer.Recognizing += (s, e) =>
{
Console.WriteLine($"\n Recognizing: {e.Result.Text}.");
};
recognizer.Recognized += (s, e) =>
{
Console.WriteLine($"\n Recognized: {e.Result.Text}.");
};
recognizer.SessionStarted += (s, e) =>
{
Console.WriteLine($"\n SessionStarted: {e.SessionId}.");
};
recognizer.SessionStopped += (s, e) =>
{
Console.WriteLine($"\n SessionStopped: {e.SessionId}.");
};
recognizer.SpeechEndDetected += (s, e) =>
{
Console.WriteLine($"\n SpeechEndDetected: {e.SessionId}.");
};
recognizer.SpeechStartDetected += (s, e) =>
{
Console.WriteLine($"\n SpeechStartDetected: {e.SessionId}.");
};
recognizer.Canceled += (s, e) =>
{
Console.WriteLine($"\n Canceled: {e.SessionId}.");
};
// Starts continuous recognition. Uses StopContinuousRecognitionAsync() to stop recognition.
Console.WriteLine("Say something...");
//await recognizer.StartContinuousRecognitionAsync().ConfigureAwait(false);
await recognizer.StartContinuousRecognitionAsync().ConfigureAwait(false);
Console.WriteLine("Press any key to stop");
Console.ReadKey();
await recognizer.StopContinuousRecognitionAsync().ConfigureAwait(false);
}
}
EDIT: After some changes, and moving the wav file locally (it was on a mapped drive), it did briefly try to run a transcription on the file, but no valid text was ever returned, only blank strings.
Transcription via microphone is working just fine. But as soon as I throw one of my .wav files at it Cognitive Services is once again crashing with the Exception code: 0xc0000094. I even tried the code that half worked, and that is also throwing the same error now.
I figured out the issue, it turned out to be the .wav files themselves. As near as I could tell, they were valid wave files. With WAV listed at the top of the binary file if you looked at it in Notepad++. However, they consistently caused Cognitive Services to crash. And the one time I got it to take one, it was unable to read it and just started running in an infinite loop returning blank strings.
I solved the issue by running the files through a double conversion. I converted them to .m4a files, then back to .wav files. Once I did that they all started working perfectly.
I originally thought it was because I was storing the files remotely on a mapped drive. However, access via mapped drive worked just fine once the files were fixed.
Hopefully Microsoft will add better error handling to the Cognitive Services wrapper. And allow the API to handle more than just wav file types.
I'm getting some trouble while running this:
public MODULEENTRY32 getModule(String ModuleName)
{
MODULEENTRY32 module32;
module32.dwSize = (uint) Marshal.SizeOf(typeof(MODULEENTRY32));
IntPtr hSnap = CreateToolhelp32Snapshot(SnapshotFlags.TH32CS_SNAPMODULE | SnapshotFlags.TH32CS_SNAPMODULE32, (uint) process.Id);
Module32First(hSnap, out module32);
if (hSnap == IntPtr.Zero)
{
return new MODULEENTRY32();
}
do
{
if (module32.szModule.Equals(ModuleName))
{
CloseHandle(hSnap);
return module32;
}
} while (Module32Next(hSnap, out module32));
return new MODULEENTRY32();
}
I was trying to get modules from a process but it always return 0,
I'm sure that the module name is corrent and the process id too
I don't think you have provided enough information to determine what the problem is.
If you read the CreateToolHelp32Snapshot documentation you should check if hSnap returned is INVALID_HANDLE_VALUE (-1). If it is, you need to call GetLastError to determine the reason for the failure.
Possible reasons for failures are documented:
If the specified process is the Idle process or one of the CSRSS
processes, this function fails and the last error code is
ERROR_ACCESS_DENIED because their access restrictions prevent
user-level code from opening them.
If the specified process is a 64-bit process and the caller is a
32-bit process, this function fails and the last error code is
ERROR_PARTIAL_COPY (299).
and:
When taking snapshots that include heaps and modules for a process
other than the current process, the CreateToolhelp32Snapshot function
can fail or return incorrect information for a variety of reasons. For
example, if the loader data table in the target process is corrupted
or not initialized, or if the module list changes during the function
call as a result of DLLs being loaded or unloaded, the function might
fail with ERROR_BAD_LENGTH or other error code. Ensure that the target
process was not started in a suspended state, and try calling the
function again. If the function fails with ERROR_BAD_LENGTH when
called with TH32CS_SNAPMODULE or TH32CS_SNAPMODULE32, call the
function again until it succeeds.
Please do not set duplicate flag on this qustion - it is not about "why ThreadAbortException occurs", it is about "why w3wp.exe process terminates after ThreadAbortException".
Let's say we have simple web application with following code sample:
protected void Page_Load(object sender, EventArgs e)
{
Response.Redirect("http://google.com");
}
Which by fact means something like (see Is Response.End() considered harmful?):
protected void Page_Load(object sender, EventArgs e)
{
...response write some data...
System.Threading.Thread.CurrentThread.Abort();
}
On my machine (Windows 10 Pro + IIS) this code leads to IIS pool process termination with error code 0x0 (redirect not performs). On other machines (which is NOT Windows 10) this code only generates ThreadAborted exception, but process continue working (redirect performs).
Can someone check this sample and explain what is going on?
UPDATE
Here some windows event logs related to this issue.
log #1
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/ROOT/AS
Process ID: 6700
Exception: System.Threading.ThreadAbortException
Message: Thread was being aborted.
StackTrace: at
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest
wr, HttpContext context) at
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr
rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData,
Int32 flags) at
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr
rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData,
Int32 flags)
log #2
Faulting application name: w3wp.exe, version: 10.0.10240.16384, time stamp: 0x559f3dad
Faulting module name: KERNELBASE.dll, version: 10.0.10240.16384, time stamp: 0x559f3b2a
Exception code: 0xe0434352
Fault offset: 0x000b3e28
Faulting process id: 0x1a2c
Faulting application start time: 0x01d0e4b1b3ed01cb
Faulting application path: C:\WINDOWS\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\WINDOWS\SYSTEM32\KERNELBASE.dll
Report Id: 23b5298d-3b36-49c7-a294-de9c864b703f
Faulting package full name:
Faulting package-relative application ID:
I was able to reproduce the issue on Server 2008r2 with .NET 4.6 installed.
I suspect it was the same problem the rest of you are running into; ThreadAbortExceptions killing the application pool in the event log (though any unhandled exception would cause the issue in my case; but that may just be a global exception handler catching it and finishing with a Response.End or Redirect). The dump stacktrace also matches the one in Ian's answer.
There was a MS Connect ticket opened for the issue, and a recent KB hotfix resolves the issue for me.
Connect Article: https://connect.microsoft.com/VisualStudio/feedback/details/1605438/disabling-ryujit-breaks-asp-net
KB Hotfix:
https://support.microsoft.com/en-us/kb/3098786
So far I have the only one solution:
static class WebExtensions
{
public static void EndSafe(this HttpResponse response)
{
response.Flush();
response.SuppressContent = true;
HttpContext.Current.ApplicationInstance.CompleteRequest();
}
public static void RedirectSafe(this HttpResponse response, string url)
{
response.Redirect(url, false);
HttpContext.Current.ApplicationInstance.CompleteRequest();
}
}
This is however forces me to ensure that there will be no code executed after it:
...some code
response.RedirectSafe(url);
return; //<-- important
...some more code
Pay attention, that only "return" is not enough in some cases (for example with recursive calls) and in some cases you may need to avoid using "return" (with try-finally constructions)
I ran into this exact same problem on Windows 8.1 today, after rebooting to install Windows Updates.
The problem was that I had manually disabled RyuJIT in the Registry, due to this issue, by adding the useLegacyJit DWORD and setting it to 1 (see Method #3). But one of the updates created the UseRyuJIT key in the same location and set it to 1 as well, and this apparently confused ASP.NET horribly.
The solution was to set useLegacyJit to 0 and issue an iisreset. After that, all is good in the world.
WinDbg's !clrstack showed the following frames when I debugged the w3wp.exe dump. Perhaps this will help others with the same error who are Googling for a solution:
000000ef9892be98 00007ffa0e2d1fea [HelperMethodFrame: 000000ef9892be98]
000000ef9892bf80 00007ff99d776588 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000ef9892df90 00007ff9fc172345 [FaultingExceptionFrame: 000000ef9892df90]
000000ef9892e490 00007ff99d7796c0 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)
000000ef9892e520 00007ff99d777377 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000ef9892e700 00007ff99d77655a System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000ef9892e740 00007ff99d775c11 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000ef9892ef58 00007ff9fc100b4e [InlinedCallFrame: 000000ef9892ef58] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000ef9892ef58 00007ff99d78cc1b [InlinedCallFrame: 000000ef9892ef58] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)
000000ef9892ef30 00007ff99d78cc1b DomainNeutralILStubClass.IL_STUB_PInvoke
000000ef9892f000 00007ff99d77756c System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)
000000ef9892f1e0 00007ff99d77655a System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)
000000ef9892f220 00007ff99d775c11 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)
000000ef9892f418 00007ff9fc100da3 [ContextTransitionFrame: 000000ef9892f418]
I've been experiencing the same issue on Win7 SP1. Web app compiled targeting .net 4.5.2 and running on .net 4.6. And I haven't been messing with the useLegacyJit or useRyuJIT registry flags.
Turned out "Enable 32-Bit applications" was unnecessarily set to Enabled on my app domain. Disabling it fixed the problem.