I have created a basic, default ASP.NET 5 project. I have a controller that creates
var engine = new V8ScriptEngine();
and returns some mock json. When I refresh page certain amount of times I get
Fatal error in heap setup
Allocation failed - process out of memory
And following stack trace
Unhandled Exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at V8Isolate.Create(StdString* , V8IsolateConstraints* , Boolean , Int32 )
at Microsoft.ClearScript.V8.V8IsolateProxyImpl..ctor(String gcName, V8RuntimeConstraints gcConstraints, Boolean enableDebugging, Int32 debugPort)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
at System.Activator.CreateInstance(Type type, Object[] args)
at Microsoft.ClearScript.V8.V8Proxy.CreateImpl[T](Object[] args)
at Microsoft.ClearScript.V8.V8IsolateProxy.Create(String name, V8RuntimeConstraints constraints, Boolean enableDebugging, Int32 debugPort)
at Microsoft.ClearScript.V8.V8Runtime..ctor(String name, V8RuntimeConstraints constraints, V8RuntimeFlags flags, Int32 debugPort)
at Microsoft.ClearScript.V8.V8ScriptEngine..ctor(V8Runtime runtime, String name, V8RuntimeConstraints constraints, V8ScriptEngineFlags flags, Int32 debugPort)
at Microsoft.ClearScript.V8.V8ScriptEngine..ctor()
I tried to look at the memory with dotMemory. Each time I refresh page, an engine is created, and adds like 2MB of ram to unmanaged memory. When it hits certain limit it crashes as explained above. As long as I click force GC before I hit the limit the memory goes down and I can use the page again.
My question is: why does GC not handle this in the first place? After each request the object can be disposed, if I force GC it does. I would asume that if I am almost out of memory but I can reclaim it with GC it would do so.
How can I resolve this? Maybe adding more memory would help but I don't know how to do this either. If GC will never clean those objects it will break anyway.
Same happens when I run Kestrel (dnx web) and with IIS.
I have framework set to "dnx46"
Here is my dnx version
$ dnx --version
Microsoft .NET Execution environment
Version: 1.0.0-rc1-16231
Type: Clr
Architecture: x86
OS Name: Windows
OS Version: 10.0
Runtime Id: win10-x86
ClearScript version is "ClearScript.V8": "5.4.3"
The short version: You need to dispose each script engine when you're done with it. A convenient way is to use the using statement:
using (var engine = new V8ScriptEngine()) {
// do stuff
}
Longer version: Each V8 instance reserves a large block of address space. These don't show up as used memory, but in a 32-bit process you can run out of address space with just a few dozen instances. The managed GC would eventually clean it all up, but because it can't track V8's address space reservations, it's in no hurry to do so, since it doesn't detect any memory pressure. Eventually you get to a point where your memory usage is still low, but V8 can no longer reserve a large-enough block of address space, and so it fails.
Related
Just recently my app has been inconsistently throwing a variety of exceptions like these:
Unable to cast object of type 'System.Threading.Thread' to type 'System.Transactions.SafeIUnknown'.
Unable to cast object of type 'System.Diagnostics.Process' to type 'System.Transactions.SafeIUnknown'.
Unable to cast object of type 'System.Drawing.SolidBrush' to type 'System.Transactions.SafeIUnknown'.
Unable to cast object of type 'System.Threading.Thread' to type 'System.Xml.Linq.XNamespace'.
(the last one might be a secondary effect of the others)
Stack traces of some of these are pasted below but note that all of them have these 3 final steps:
at System.Transactions.Transaction.JitSafeGetContextTransaction(ContextData contextData)
at System.Transactions.Transaction.FastGetTransaction(TransactionScope currentScope, ContextData contextData, Transaction& contextTransaction)
at System.Transactions.Transaction.get_Current()
Observations - these exceptions:
Occur frequently, but not all the time (50%?)
Don't always produce the same exception when they do occur
Have no obvious connection to the code in my app which leads to them
Are reproducible on various PCs
Can't be found exactly in any searches I've done
The app is a set of C# DLLs called via COM from older VB6 code. However that has been in place for years, and we haven't changed anything about how that is done recently.
It is not a multi-threaded app, so the casts involving System.Threading.Thread seem out of place. There was one third party module that used threads, but we took it out as an experiment: doing so did not resolve the problem but seemed to reduce the frequency of errors. This leads me to think its some kind of resource clobbering problem.
I am looking for advice such as:
If you've actually seen this pattern of exceptions before, what was the cause?
Recommendations for how to go about debugging the root issue
Example stack traces:
Unable to cast object of type 'System.Threading.Thread' to type 'System.Transactions.SafeIUnknown'.
at System.Transactions.Transaction.JitSafeGetContextTransaction(ContextData contextData)
at System.Transactions.Transaction.FastGetTransaction(TransactionScope currentScope, ContextData contextData, Transaction& contextTransaction)
at System.Transactions.Transaction.get_Current()
at System.Data.SQLite.SQLiteConnection.Open()
at Company.ABC.WrappedDbConnection.Open()
at Company.ABC.Program.DoSomething()
and
Unable to cast object of type 'System.Threading.Thread' to type 'System.Transactions.SafeIUnknown'.
at System.Transactions.Transaction.JitSafeGetContextTransaction(ContextData contextData)
at System.Transactions.Transaction.FastGetTransaction(TransactionScope currentScope, ContextData contextData, Transaction& contextTransaction)
at System.Transactions.Transaction.get_Current()
at System.Data.Common.ADP.IsSysTxEqualSysEsTransaction()
at System.Data.Common.ADP.NeedManualEnlistment()
at System.Data.OleDb.OleDbConnection.Open()
at Company.ABC.WrappedDbConnection.Open()
at Company.ABC.Program.DoSomething()
and
System.InvalidCastException: Unable to cast object of type 'System.Drawing.SolidBrush' to type 'System.Transactions.SafeIUnknown'.
at System.Transactions.Transaction.JitSafeGetContextTransaction(ContextData contextData)
at System.Transactions.Transaction.FastGetTransaction(TransactionScope currentScope, ContextData contextData, Transaction& contextTransaction)
at System.Transactions.Transaction.get_Current()
at System.Data.Common.ADP.IsSysTxEqualSysEsTransaction()
at System.Data.Common.ADP.NeedManualEnlistment()
at System.Data.OleDb.OleDbConnection.Open()
at Company.ABC.WrappedDbConnection.Open()
at Company.ABC.Program.DoSomething()
EDIT:
I tried looking into this using windbg (with SOS extensions for managed code).
It was able to break on the exception. Here is the stack trace from that point:
0:000> !DumpStack
OS Thread Id: 0x2d44 (0)
Current frame: KERNELBASE!RaiseException+0x62
ChildEBP RetAddr Caller, Callee
0019b658 750325f2 KERNELBASE!RaiseException+0x62, calling ntdll!RtlRaiseException
0019b688 774a7310 ntdll!RtlFreeHeap+0x1e0, calling ntdll!RtlFreeHeap+0x560
0019b69c 70a9f09f clr+0xf09f, calling KERNEL32!TlsGetValue
0019b6ac 70b928f1 clr!GetMetaDataPublicInterfaceFromInternal+0x9741, calling KERNELBASE!RaiseException
0019b6ec 70aae56d clr!LogHelp_NoGuiOnAssert+0x2add, calling clr!LogHelp_NoGuiOnAssert+0x2a92
0019b6f8 70aae50f clr!LogHelp_NoGuiOnAssert+0x2a7f
0019b748 70bfc1a2 clr!CreateApplicationContext+0x1c2d2, calling clr!GetMetaDataPublicInterfaceFromInternal+0x95b6
0019b77c 70e3d369 clr!CreateHistoryReader+0x72069, calling clr!CreateApplicationContext+0x1c279
0019b7dc 70bc6a59 clr!PreBindAssemblyEx+0x10959, calling clr+0xf7b0
0019b80c 09364844 (MethodDesc 08e0d5b4 +0x4c System.Transactions.Transaction.JitSafeGetContextTransaction(System.Transactions.ContextData)), calling clr!LogHelp_TerminateOnAssert+0x7f00
0019b83c 09362f2f (MethodDesc 08e0d5e4 +0x9f System.Transactions.Transaction.FastGetTransaction(System.Transactions.TransactionScope, System.Transactions.ContextData, System.Transactions.Transaction ByRef)), calling (MethodDesc 08e0d5b4 +0 System.Transactions.Transaction.JitSafeGetContextTransaction(System.Transactions.ContextData))
0019b850 09362510 (MethodDesc 08e0d5fc +0x78 System.Transactions.Transaction.get_Current()), calling (MethodDesc 08e0d5e4 +0 System.Transactions.Transaction.FastGetTransaction(System.Transactions.TransactionScope, System.Transactions.ContextData, System.Transactions.Transaction ByRef))
0019b86c 08ddf67a (MethodDesc 08e04c6c +0x14b2 System.Data.SQLite.SQLiteConnection.Open()), calling (MethodDesc 08e0d5fc +0 System.Transactions.Transaction.get_Current())
0019ba70 08ddd8ea (MethodDesc 08e058b0 +0x22 Company.ABC.WrappedDbConnection.Open()), calling 0dfce43a
This syncs with the CLR exception stacks that are above.
The thing I was mainly looking to ascertain is whether there is heap "corruption". But apparently not:
0:000> !VerifyHeap
No heap corruption detected.
So I think that rules out any of our native code (or 3rd party) simply wreaking havoc. But still no actual understanding of this problem. It looks more like a bug in the database layer to me.
This has been submitted as an issue here: https://developercommunity2.visualstudio.com/t/Unable-to-cast-object-of-type-varies/1241378
I don't think this is a perfect "root cause" answer; but it is the workaround which seems to have succeeded in avoiding the exceptions.
The code in question is used to create DB connection, execute a query, then teardown the connection.
The exact change was to go from:
public class DatabaseAccessXYZ
{
IDbConnection connection;
void Execute<T>(string sql, T item)
{
using (connection = ConnectionManagerXYZ.GetConnection(Filespec))
{
connection.Open();
connection.Execute(sql, item);
}
...
to this instead:
public class DatabaseAccessXYZ
{
void Execute<T>(string sql, T item)
{
using (var connection = ConnectionManagerXYZ.GetConnection(Filespec))
{
connection.Open();
connection.Execute(sql, item);
}
...
So the scope of the connection variable was changed. It was only used in this one place anyway, it did not need to be a class-level field. Now the exception no longer occurs.
As far as I understand it, this change just means that the CLR will probably garbage collect the object sooner. But it didn’t affect anything else – for instance, the connection is still closed at the same time (right after Execute is called when the using completes and the connection is disposed) which I confirmed in the debugger.
As noted in the question since there is no heap corruption I suspect that the underlying issue is a bug, maybe in System.Transactions. Databases of different types were in use (SQLite and MS Access / Jet) so I don't think the issue could be in the actual connection objects, which would have had to differ.
I'm trying to create an instance of System.RuntimeType which has a very annoying constructor:
internal class RuntimeType
{
internal RuntimeType()
{
throw new NotSupportedException();
}
}
Normally you could just try/catch it and the object will have been constructed, even though the exception was thrown. However, being internal, I have to use reflection to call these, and the exception will get wrapped into a TargetInvocationException and the object thrown away:
System.Reflection.TargetInvocationException : Exception has been thrown by the target of an invocation.
----> System.NotSupportedException : Specified method is not supported.
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.ConstructorInfo.Invoke(Object[] parameters)
at Application.Util.Constructor.RuntimeType() in C:\Work\Application\src\Application\Util\Constructor.cs:line 300
at Tests._4._5.Util.ConstructorTests.RuntimeType() in C:\Work\Application\src\Tests\Tests.4.5\Util\ConstructorTests.cs:line 159
--NotSupportedException
at System.RuntimeType..ctor()
System.RuntimeMethodHandle.InvokeMethod() is marked extern so I can't see how exactly it works to reverse-engineer it without the wrapping. Is there any way to get this type constructed?
The FormatterServices.GetUninitializedObject method can create objects without invoke a constructor:
Return Type: System.Object
A zeroed object of the specified type.
https://msdn.microsoft.com/en-us/library/system.runtime.serialization.formatterservices.getuninitializedobject(v=vs.110).aspx
I am trying to produce my own Test Runner that accesses unit test methods via reflection and runs them inside a separate AppDomain.
object instance = Activator.CreateInstance(targetType);
foreach (MethodInfo method in methods)
{
method.Invoke(instance, null);
}
This works perfectly until one of the tests uses Microsoft.Fakes at which point when the method attempts to create the ShimsContext it throws this exception:
System.Reflection.TargetInvocationException was unhandled
HResult=-2146232828
Message=Exception has been thrown by the target of an invocation.
Source=mscorlib
StackTrace:
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
...
InnerException:
HResult=-2146233088
Message=Failed to resolve profiler path from COR_PROFILER_PATH and COR_PROFILER environment variables.
Source=Microsoft.QualityTools.Testing.Fakes
StackTrace:
at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.ResolveProfilerPath()
at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.IntelliTraceInstrumentationProvider.Initialize()
at Microsoft.QualityTools.Testing.Fakes.UnitTestIsolation.UnitTestIsolationRuntime.InitializeUnitTestIsolationInstrumentationProvider()
at Microsoft.QualityTools.Testing.Fakes.Shims.ShimRuntime.CreateContext()
at Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create()
at UnitTests.TestRunnerTests.Builders.AppDomainBuilderTests.AppDomainBuilderBuildsAppDomainsWithPermissionStateNone() in c:\VS\UnitTests\TestRunnerTests\Builders\AppDomainBuilderTests.cs:line 130
InnerException:
While trying to run this test, I am able to successfully call the TestInitialise Method, but it throws this exception during the TestMethod which looks like this...
/// <summary>
/// A test to ensure that the Application domain builder builds an
/// application domains with permission state none.
/// </summary>
[TestMethod]
public void AppDomainBuilderBuildsAppDomainsWithPermissionStateNone()
{
// Arrange
using (ShimsContext.Create())
{
bool wasCalledCorrectly = false;
PermissionState expected = PermissionState.None;
ShimPermissionSet.ConstructorPermissionState = (permisionSet, actual) =>
{
if (expected == actual)
{
wasCalledCorrectly = true;
}
};
AppDomainBuilder target = new AppDomainBuilder();
// Act
target.Build(this.defaultInput);
// Assert
Assert.IsTrue(wasCalledCorrectly);
}
}
Having looked into the exception details, it appears to be because the tests can't access the Intellisense Profiler via the environment variables. I was wondering if anybody would have any experience in injecting this object into the AppDomain so that the Fakes can be successfully generated?
Or do I need to include the VSTest.ExecutionEngine in the AppDomain and invoke it somehow?
Any help on shining some light on this would be highly appreciated as being able to support Microsoft.Fakes without having to call out to the VSTest.Console.Exe would be highly desirable.
Thanks
I have developed a web application with ASP.net(C#, .Net Framework 4.0) in some part of the application I am calling an API to get some information. I noticed that some of the calls to API are failed and I got this error:
Index was outside the bounds of the array.
when I checked the stack trace I saw this :
at System.Collections.Generic.Dictionary2.Insert(TKey key, TValue value, Boolean add)
at System.Collections.Generic.Dictionary2.set_Item(TKey key, TValue
value)
at Navitaire.Ncl.Validation.ValidationManager.getValidationAttributes(MemberInfo mi, Boolean& skip)
at Navitaire.Ncl.Validation.ValidationManager.validate(Object declaringObject, Object value, MemberInfo mi, List1 results)
at Navitaire.Ncl.Validation.ValidationManager.Validate(Object obj)
at Navitaire.Ncl.ServiceModel.ParameterValidationInspector.BeforeCall(String operationName, Object[] inputs)
at Navitaire.Ncl.ServiceModel.Remoting.BoilerplateSinkBase.InvokeBeforeCallInspectors(StateData stateData, RemotingMessage& message)
at Navitaire.Ncl.ServiceModel.Remoting.BoilerplateServerSink.ProcessMessage(IServerChannelSinkStacksinkStack, IMessage requestMsg, ITransportHeaders requestHeaders, Stream requestStream, IMessage& responseMsg, ITransportHeaders&
responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage(IServerChannelSinkStack sinkStack, IMessage requestMsg, ITransportHeaders requestHeaders, Stream requestStream, IMessage& responseMsg, ITransportHeaders&
responseHeaders, Stream& responseStream)
I put a break point on the catch exception part, so every time that I got the error, application will stop and I can read the exception. I noticed that if in this moment I call the API again (move the cursor to the API call line) it works fine. so it means there is no problem with parameter that I am passing to the API.
I tried to run the application on three different environments with totally different network and different internet connection, but still I got the same error.
Can you please someone help me on this case?
Thanks
I suspect this is a problem with the API, rather than your code, especially if it works the second time you make the same call.
I believe the API is doing something with an internal dictionary, that doesn't quite get set up correctly the first time you call it, but by the second time, it has been and works "correctly".
Are you able to view / have access to the code for the API method your calling?
If so, you are best running that in debug alongside your application, placing a breakpoint on the entry to that method and stepping over the lines within it to see what it is doing and why it may be breaking.
If it's your API, or you're allowed to share the code of this method and you can't find the error yourself, please post up the code of the API method and I'll help you find what the cause could be.
If you are not able to view the code, I would suggest contacting the producer of the API if it's a private one, or if it's an open source API, share the link and provide a code sample of what your code looks like (showing how you make the API call) and we can help diagnose it.
I checked with the API developer and I found that there was some thing wrong on their side after they fixed the bug I did not get any error anymore.
We have a .net 3.5 multi-threaded windows forms application. Is is launching 4-5 background workers/ async calls to retrieve data. About 1 in 10 times, one of the threads throws the following error. Sometimes the error is a null exception instead but the same exact call stack. We have tracked the issue down to the thread somehow losing its CultureInfo. Anyone else ran into this?
System.Threading.SynchronizationLockException: Object synchronization method was called from an unsynchronized block of code.
at System.Globalization.CultureTableRecord.GetCultureTableRecord(String name, Boolean useUserOverride)
at System.Globalization.CultureTableRecord.GetCultureTableRecord(Int32 cultureId, Boolean useUserOverride)
at System.Globalization.CultureInfo..ctor(Int32 culture, Boolean useUserOverride)
at System.Globalization.CultureInfo.GetCultureByLCIDOrName(Int32 preferLCID, String fallbackToString)
at System.Globalization.CultureInfo.InitUserDefaultUICulture()
at System.Globalization.CultureInfo.get_UserDefaultUICulture()
at System.Threading.Thread.get_CurrentUICulture()
at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
at System.Environment.ResourceHelper.GetResourceStringCode(Object userDataIn)
at System.Environment.GetResourceFromDefault(String key)
at System.Diagnostics.StackTrace.ToString(TraceFormat traceFormat)
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.IO.StreamWriter.Init(Stream stream, Encoding encoding, Int32 bufferSize)
at System.IO.StreamWriter..ctor(Stream stream, Encoding encoding, Int32 bufferSize)
at System.Web.Services.Protocols.SoapHttpClientProtocol.GetWriterForMessage(SoapClientMessage message, Int32 bufferSize)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Serialize(SoapClientMessage message)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
Try playing with default culture for entire domain using CultureInfo.DefaultThreadCurrentCulture and/or CultureInfo.DefaultThreadCurrentUICulture and see if that helps. All in all, you'll be able to ensure whether it's related to thread's culture
UPD: that specific exception may be thrown when using sync methods like lock, Monitor, etc on value types, calls to sync methods will box value, creating new one every time. I doubt it could be such serios bug in framework.
Can you check your synchronization code?
UPD2: ok, let's try to debug.
First off, turn on break on every exception thrown
Next, enable .NET Framework debugging that will help you to
locate an error
After that, there is a high chance for a workaround/solution for you problem
From just looking at the call stack, this appears to be a framework bug (of sorts). It seems that the StreamWriter constructor is performing a non-thread-safe operation (accessing GetCultureTableRecord) through a complex series of calls. My recommendation would be to serialize your calls to the StreamWriter constructor:
// Put this somewhere convenient
public static Object swConstructorLock = new Object();
then, when you create your StreamWriters:
// Lock this constructor because it does something that isnt thread-safe as per http://stackoverflow.com/questions/16113366/windows-forms-threads-are-losing-their-culture
StreamWriter myStreamWriter;
lock (swConstructorLock) {
myStreamWriter = new StreamWriter(theStreamToWrite);
}
I found that CultureInfo.ClearCachedData was being called on every web request :/ Removing that fixed the issue. There seems that Culture still needs some sort of lock since thread A calls ClearCachedData and thread B requests the data and it throws a null exception.
I got this exception (once, out of a blue), also in a .NET 3.5 app:
System.Threading.SynchronizationLockException: Object synchronization method was called from an unsynchronized block of code.
at System.Globalization.CultureTableRecord.GetCultureTableRecord(String name, Boolean useUserOverride)
at System.Globalization.CultureTableRecord.GetCultureTableRecord(Int32 cultureId, Boolean useUserOverride)
at System.Globalization.CultureInfo..ctor(Int32 culture, Boolean useUserOverride)
at System.Globalization.CultureInfo.GetCultureByLCIDOrName(Int32 preferLCID, String fallbackToString)
at System.Globalization.CultureInfo.InitUserDefaultCulture()
at System.Globalization.CultureInfo.get_UserDefaultCulture()
at System.Threading.Thread.get_CurrentCulture()
at System.Globalization.DateTimeFormatInfo.get_CurrentInfo()
at System.DateTime.ToString(String format)
This happened in a locked section, but the lock was not a value type.