I try to build a little tensorflow application with TensorFlowSharp and sometimes I recieve this exception:
Managed Debugging Assistant 'CallbackOnCollectedDelegate'
For the function TensorFlowSharp!TensorFlow.TFBuffer+BufferReleaseFunc::Invoke
I tried to find out what it means but I didn't fully understand the explanations. This is the part of the code where the exception is thrown:
var graph = new TFGraph();
var model = File.ReadAllBytes(ModelsFile);
graph.Import(model, "");
Does somebody know what I should do to prevent this exception?
Bruno
I assume this is a bug in TensorflowSharp.
The error looks like a usually inconsistent access violation in the CLR code (occurs usually only under heavy load or on a random number of attempts). Citing from Microsoft docs:
The callbackOnCollectedDelegate managed debugging assistant (MDA) is
activated if a delegate is marshaled from managed to unmanaged code as
a function pointer and a callback is placed on that function pointer
after the delegate has been garbage collected.
This type of error occurs when a delegate from which the function pointer was created and exposed to unmanaged code was garbage collected. When the unmanaged component tries to call on the function pointer, it generates an access violation. The failure appears random because it depends on when garbage collection occurs.
The resolution can be difficult, since once a delegate has been marshaled out as an unmanaged function pointer, the garbage collector cannot track its lifetime. Instead, it is required to keep a reference to the delegate for the lifetime of the unmanaged function pointer. In order to do this, the faulty delegate that was collected, has to be identified in TensorFlowShapr's code (or your code).
You can also enable the gcUnmanagedToManaged MDA to force a garbage
collection before every callback into the runtime. This will remove
the uncertainty introduced by the garbage collection by ensuring that
a garbage collection always occurs before the callback. Once you know
what delegate was collected, change your code to keep a reference to
that delegate on the managed side for the lifetime of the marshaled
unmanaged function pointer.
So, I guess it's best to report this to the maker of the library.
I see only one good candidate for this bug, a delegate in Buffer.cs. But Miguel already fixed this bug on July 27th, the diff is here. So just make sure to update your copy. If you obtained it from Nuget then ensure you have at least version 1.30
Related
I'm running into a problem where it would appear the GC thread is waking up and deleting an object while it's in use.
While processfoo is running, and before it returns, it would appear fooCopy destructor fires in a separate thread. Has anyone seen a problem like this and if so how do you work around it? Surely, this can't be really happening and I must be doing something wrong. Can anyone give me some tips/strategies to debug the garbage collection?
try
{
CFoo fooCopy = new CFoo(someFoo);
someBar.ProcessFoo(fooCopy);
}
CFoo has an IntPtr member variable.
ProcessFoo passes that member variable to an imported function from a C++ DLL that might take a few seconds to run.
In my C++ DLL I log when the IntPtr is created and when it is deleted and I can see a separate thread deleting the IntPtr while ProcessFoo is running.
Surely, this can't be really happening and I must be doing something wrong.
Nope, it can absolutely be happening (this is why writing finalizers is hard, and you should avoid them where you can, and be very careful when forced to write them). An object can be GC-ed as soon as the runtime can prove that no code will ever try to access it again. If you call an instance method (and never access the object instance any time after that invocation) then the object is eligible for collection immediately after the last usage of this from that instance method.
Objects that are in scope (say, for example, the this variable of a method) but that are never used again within that scope are not considered rooted by the GC.
As for how you work around it, if you have a variable that you want to be considered "alive" even though it's never accessed in managed code again, use GC.KeepAlive. In this case, adding GC.KeepAlive(this) to the end of ProcessFoo will ensure that the object in question stays alive until the end of the method (or if it's not that method's responsibility, have the caller call GC.KeepAlive(someBar) right after ProcessFoo).
See this blog post for more information on this topic, and a few related and even more unusual properties of finalizers.
This is pretty normal when you interop with C++, the GC has no hope of discovering that the IntPtr is in use anywhere else. It is not a reference type that the GC has awareness of, nor can it probe the stack frames of native code. The jitter marks the fooCopy object reference in use up to the underlying CALL, not beyond that. In other words, it is eligible for collection while the native code is executing. If another thread triggers a GC then it is sayonora.
You'll find details about the lifetime of local variables in this post.
There are several possible workarounds for this, albeit that the correct one can't be guessed from the question. Beyond the SafeHandle classes, very good at ensuring the finalization is taken care of as well, you could use HandleRef instead of IntPtr in the [DllImport] declaration. Or append GC.KeepAlive() to this code to force the jitter to extend the lifetime of fooCopy.
There are lots of questions on SO regarding the releasing COM objects and garbage collection but nothing I could find that address this question specifically.
When releasing COM objects (specifically Excel Interop in this case), in what order should I be releasing the reference and calling garbage collection?
In some places (such as here) I have seen this:
Marshall.FinalReleaseComObject(obj);
GC.Collect();
GC.WaitForPendingFinalizers();
And in others (such as here) this:
GC.Collect();
GC.WaitForPendingFinalizers();
Marshall.FinalReleaseComObject(obj);
Or doesn't it matter and I'm worrying about nothing?
Marshal.FinalReleaseComObject() releases the underlying COM interface pointer.
GC.Collect() and GC.WaitForPendingFinalizers() causes the finalizer for a COM wrapper to be called, which calls FinalReleaseComObject().
So what makes no sense is to do it both ways. Pick one or the other.
The trouble with explicitly calling FinalReleaseComObject() is that it will only work when you call it for all the interface pointers. The Office program will keep running if you miss just one of them. That's very easy to do, especially the syntax sugar allowed in C# version 4 makes it likely. An expression like range = sheet.Cells[1, 1], very common in Excel interop code. There's a hidden Range interface reference there that you never explicitly store anywhere. So you can't release it either.
That's not a problem with GC.Collect(), it can see them. It is however not entirely without trouble either, it will only collect and run the finalizer when your program has no reference to the interface anymore. Which is definitely what's wrong with your second snippet. And which tends to go wrong when you debug your program, the debugger extends the lifetime of local object references to the end of the method. Also the time you look at Taskmgr and yell "die dammit!"
The usual advice for GC.Collect() applies here as well. Keep your program running and perform work. The normal thing happens, you'll trigger a garbage collection and that releases the COM wrappers as well. And the Office program will exit. It just doesn't happen instantly, it will happen eventually.
Reference counting mechanism that is used by COM is another way of automatic memory management but with slightly different impact on memory and behavior.
Any reference counting implementation provide deterministic behavior for resource cleanup. This means that right after call to Marshal.FinalReleaseComObject() all resources (memory and other resources) related to the COM object would be reclaimed.
This means that if we have additional managed objects and you want to reclaim them as quickly as possible, you should release COM object first and only after that call GC.Collect method.
I have an managed .NET 2.0 dll that provides an IDispatch inteface and calls an IDispatch interface. Reading this question this question:
When to use ReleaseComObject vs FinalReleaseComObject?
It sounds as though I should be waiting for Garbage Collection to release my COM object references rather than attempting to count them myself. However when I do this, the unmanaged code calling my DLL throws an error (I don't have the source code to debug what the error is) but it indicates it's an unauthorized virtual memory access problem. Adding the FinalReleaseCOMObject stops this error.
Should I be using FinalReleaseCOMObject then?
EDIT: forgot to mention when the managed code is replaced by an unmanaged dll it does not exhibit the error.
I'm currently looking into using C++/CLI to bridge the gap between managed C# and native, unmanaged C++ code. One particular issue I'm looking to resolve, is the conversion of data types that are different in C# and C++.
While reading up on the use of such a bridging approach and the performance implications involved, I wondered how Garbage Collection would work. Specifically, how the Garbage Collector would handle cleanup of objects created on either side, if they are referenced / destroyed on the 'other side'.
So far, I've read various articles and forum questions on StackOverflow and MSDN, which has lead me to believe that the Garbage Collector should work across both types of code when running in the same process - i.e if an object was created in C# and passed to the C++/CLI bridge, it would not be collected until the references on both sides were no longer in use.
My question in this case, breaks down into three parts:
Am I right in concluding that the Garbage Collector works across both portions of code (C# and C++/CLI) when running in the same process?
In relation to 1: how does it work in such a circumstance (specifically in terms of cleaning up objects referenced by both code bases).
Are there any suggestions on how to monitor the activity of the Garbage Collector - i.e. writing tests to check when Garbage Collection occurs; or a program that monitors the Garbage Collector itself.
I already have somewhat of an understanding of how the Garbage Collector works in general, so my questions here are specific to the following scenario:
Components
Assembly A - (written in C#)
Assembly B - (written in C++/CLI)
Program Execution
Object O is created in Assembly A.
Object O is passed into a function inside Assembly B.
Reference to object O in Assembly A is released.
Assembly B holds onto reference to object O.
Execution ends (i.e. via program exit).
Assembly B releases reference to object O.
Thanks in advance for any thoughts on this question. Let me know if further information is needed or if something is not clear enough.
EDIT
As per request, I have written a rough example of the scenario I'm trying to describe. The C# and C++/CLI code can be found on PasteBin.
When the code is actually running, none of it will be C# or C++/CLI. All of it will be IL from the C# and C++/CLI and machine code from the native code you're interoperating with.
Hence you could re-write part of your question as:
Assembly A - (IL and we don't know what it was written in)
Assembly B - (IL and we don't know what it was written in)
Of the managed objects, all of them will be garbage collected as per the same rules, unless you use a mechanism to prevent it (GC.KeepAlive). All of them could be moved in memory unless you pin them (because you're passing addresses to unmanaged code.
.NET Profiler will give you some information on garbage collection, as will the collection counts in performance monitor.
Am I right in concluding that the Garbage Collector works across both
portions of code (C# and C++/CLI) when running in the same process?
Yes, a single garbage collector works inside one process for both (C# and Managed C++) . If inside one process there is code that is running under different CLR versions then there will be different instance of GC for each CLR version
In relation to 1: how does it work in such a circumstance (specifically
in terms of cleaning up objects referenced by both code bases).
I think for GC it doesn't matter if the code is managed C# or C++/CLI (Note that GC will only manage C# and Managed C++ code not native C++). It will work in it's own way without considering what type of code underlying is. Regarding freeing memory, GC will do that whenever an object can no longer be referenced. So as long as there is something referring to a variable it won't be collected regardless of assembly. For native C++ code you will have to manually free every dynamically allocated memory
Are there any suggestions on how to monitor the activity of the Garbage
Collector - i.e. writing tests to check when Garbage Collection occurs; or
a program that monitors the Garbage Collector itself.
You can use tools like .Net Profiler for monitoring. Also take a look at Garbage Collection Notifications
Will using GetComInterfaceForObject and passing the returned IntPtr to unmanaged code keep the managed object from being moved in memory? Or does the clr somehow maintain that ptr? Note that the unmanaged code will use this for the lifetime of the program, and I need to make sure the managed object is not being moved by the GC.(At least I think that's right?)
EDIT - Alright I found some info and I am thinking that this may be the answer. It deals with delegates, but I would have to believe calling GetComInterfaceForObject does something along the same lines.
Source of the Following text
"Managed Delegates can be marshaled to unmanaged code,
where they are exposed as unmanaged function pointers. Calls on those
pointers will perform an unmanaged to managed transition; a change in
calling convention; entry into the correct AppDomain; and any necessary
argument marshaling. Clearly the unmanaged function pointer must refer to a
fixed address. It would be a disaster if the GC were relocating that! This
leads many applications to create a pinning handle for the delegate. This
is completely unnecessary. The unmanaged function pointer actually refers
to a native code stub that we dynamically generate to perform the transition
& marshaling. This stub exists in fixed memory outside of the GC heap.
However, the application is responsible for somehow extending the lifetime
of the delegate until no more calls will occur from unmanaged code. The
lifetime of the native code stub is directly related to the lifetime of the
delegate. Once the delegate is collected, subsequent calls via the
unmanaged function pointer will crash or otherwise corrupt the process. In
our recent release, we added a Customer Debug Probe which allows you to
cleanly detect this all too common bug in your code. If you havent
started using Customer Debug Probes during development, please take a look!"
As your edit states (about delegates), your managed object doesn't need to be pinned, since GetComInterfaceForObject returns a "pinned" pointer that calls through to the correct managed object. However, you will need to make sure that the managed object lives for as long as the COM clients are using the unmanaged pointer to it.