Develop Reference

Where do I put GC.KeepAlive?

I need to register a callback in unmanaged code, but it looks like GC keeps collecting my reference. So I added GC.KeepAlive(callback_pin); but it has no effect. I'm not sure where should I put GC.KeepAlive.
This is the code where I register my own callback to the unmanaged event, it is called from a thread. (Some_Callback and Some_Method are external objects)
var callback_pin = new Some_Callback(MyManagedCallback);
GC.KeepAlive(callback_pin);
Some_Method(callback_pin);
return true;
And below is how I imported the unmanaged code. The documentation which came with it suggests that I use the above-mentioned code in order to keep the callback alive, but since the callback is never fired in my case I don't think it's the right way to do it. Any enlightenments?
[UnmanagedFunctionPointer(CallingConvention.StdCall, CharSet = CharSet.Unicode)]
public delegate void Some_Callback(Int32 line, [MarshalAs(UnmanagedType.LPWStr)] string msg);
[DllImport("SOME_DLL.dll", CharSet = CharSet.Unicode)]
public static extern Int32 Some_Method(MulticastDelegate funcPtr, Int32 mask);

Delegates are automatically "kept alive" (the technical term is "rooted") for the duration of native functions in which they participate as parameters.
You only need special code to keep them alive if the native function you're calling is only storing the pointer, then at some later point another function (or thread) uses the stored pointer. The framework could conceivably garbage collect your delegate by then.
Besides, your issue is different, if the delegate was garbage collected you'd be getting an access violation when calling it. If nothing is happening, your native function simply isn't calling it -- debug it!

Delegate on instance method passed by P/Invoke

To my surprise I have discovered a powerful feature today. Since it looks too good to be true I want to make sure that it is not just working due to some weird coincidence.
I have always thought that when my p/invoke (to c/c++ library) call expects a (callback) function pointer, I would have to pass a delegate on a static c# function. For example in the following I would always reference a delegate of KINSysFn to a static function of that signature.
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate int KINSysFn(IntPtr uu, IntPtr fval, IntPtr user_data );
and call my P/Invoke with this delegate argument:
[DllImport("some.dll", EntryPoint = "KINInit", ExactSpelling = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int KINInit(IntPtr kinmem, KINSysFn func, IntPtr tmpl);
But now I just tried and passed a delegate on an instance method and it worked also! For example:
public class MySystemFunctor
{
double c = 3.0;
public int SystemFunction(IntPtr u, IntPtr v, IntPtr userData) {}
}
// ...
var myFunctor = new MySystemFunctor();
KINInit(kinmem, myFunctor.SystemFunction, IntPtr.Zero);
Of course, I understand that inside managed code there is no technical problem at all with packaging a "this" object together with an instance method to form the respective delegate.
But what surprises me about it, is the fact that the "this" object of MySystemFunctor.SystemFunction finds its way also to the native dll, which only accepts a static function and does not incorporate any facility for a "this" object or packaging it together with the function.
Does this mean that any such delegate is translated (marshalled?) individually to a static function where the reference to the respective "this" object is somehow hard coded inside the function definition? How else could be distinguished between the different delegate instances, for example if I have
var myFunctor01 = new MySystemFunctor();
// ...
var myFunctor99 = new MySystemFunctor();
KINInit(kinmem, myFunctor01.SystemFunction, IntPtr.Zero);
// ...
KINInit(kinmem, myFunctor99.SystemFunction, IntPtr.Zero);
These can't all point to the same function. And what if I create an indefinite number of MySystemFunctor objects dynamically? Is every such delegate "unrolled"/compiled to its own static function definition at runtime?

Does this mean that any such delegate is translated (marshalled?) individually to a static function...
Yes, you guessed at this correctly. Not exactly a "static function", there is a mountain of code inside the CLR that performs this magic. It auto-generates machine code for a thunk that adapts the call from native code to managed code. The native code gets a function pointer to that thunk. The argument values may have to be converted, a standard pinvoke marshaller duty. And always shuffled around to match the call to the managed method. Digging up the stored delegate's Target property to provide this is part of that. And it jiggers the stack frame, tying a link to the previous managed frame, so the GC can see that it again needs to look for object roots.
There is however one nasty little detail that gets just about everybody in trouble. These thunks are automatically cleaned-up again when the callback is not necessary anymore. The CLR gets no help from the native code to determine this, it happens when the delegate object gets garbage-collected. Maybe you smell the rat, what determines in your program when that happens?
var myFunctor = new MySystemFunctor();
That is a local variable of a method. It is not going to survive for very long, the next collection will destroy it. Bad news if the native code keeps making callbacks through the thunk, it won't be around anymore and that's a hard crash. Not so easy to see when you are experimenting with code since it takes a while.
You have to ensure this can't happen. Storing the delegate objects in your class might work, but then you have to make sure your class object survives long enough. Whatever it takes, no guess from the snippet. It tends to solve itself when you also ensure that you unregister these callbacks again since that requires storing the object reference for use later. You can also store them in a static variable or use GCHandle.Alloc(), but that of course loses the benefit of having an instance callback in a hurry. Feel good about having this done correctly by testing it, call GC.Collect() in the caller.
Worth noting is that you did it right by new-ing the delegate explicitly. C# syntax sugar does not require that, makes it harder to get this right. If the callbacks only occur while you make the pinvoke call into the native code, not uncommon (like EnumWindows), then you don't have to worry about it since the pinvoke marshaller ensures the delegate object stays referenced.

For the records: I have walked right into the trap, Hans Passant has mentioned. Forced garbage collection has led to a null reference exception because the delegate was transient:
KINInit(kinmem, myFunctor.SystemFunction, IntPtr.Zero);
// BTW: same with:
// KINInit(kinmem, new KINSysFn(myFunctor.SystemFunction), IntPtr.Zero);
GC.Collect();
GC.WaitForPendingFinalizers();
KINSol(/*...*); // BAAM! NullReferenceException
Luckily I had already wrapped the critical two P/Invokes, KINInit (which sets the callback delegate) and KINSolve (which actually uses the callback) into a dedicated managed class. The solution was, as already discussed, to keep the delegate referenced by a class member:
// ksf is a class member of delegate type KINSysFn that keeps ref to delegate instance
ksf = new KINSysFn(myFunctor.SystemFunction);
KINInit(kinmem, ksf, IntPtr.Zero);
GC.Collect();
GC.WaitForPendingFinalizers();
KINSol(/*...*);
Thanks again, Hans, I'd never have noticed this flaw because it works as long as no GC happens!

Where function passed as UnmanagedFunctionPointer to C is executed?

In C, there is a function that accepts a pointer to a function to perform comparison:
[DllImport("mylibrary.dll", CallingConvention = CallingConvention.Cdecl)]
private static extern int set_compare(IntPtr id, MarshalAs(UnmanagedType.FunctionPtr)]CompareFunction cmp);
In C#, a delegate is passed to the C function:
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
delegate int CompareFunction(ref IntPtr left, ref IntPtr right);
Currently, I accept Func<T,T,int> comparer in a constructor of a generic class and convert it to the delegate. "mylibrary.dll" owns data, managed C# library knows how to convert pointers to T and then compare Ts.
//.in ctor
CompareFunction cmpFunc = (ref IntPtr left, ref IntPtr right) => {
var l = GenericFromPointer<T>(left);
var r = GenericFromPointer<T>(right);
return comparer(l, r);
};
I also have an option to write a CompareFunction in C for most important data types that are used in 90%+ cases, but I hope to avoid modifications to the native library.
The question is, when setting the compare function with P/Invoke, does every subsequent call to that function from C code incurs marshaling overheads, or the delegate is called from C as if it was initially written in C?
I imagine that, when compiled, the delegate is a sequence of machine instructions in memory, but do not understand if/why C code would need to ask .NET to make the actual comparison, instead of just executing these instructions in place?
I am mostly interested in better understanding how interop works. However, this delegate is used for binary search on big data sets, and if every subsequent call has some overheads as a single P/Invoke, rewriting comparers in native C could be a good option.

I imagine that, when compiled, the delegate is a sequence of machine instructions in memory, but do not understand if/why C code would need to ask .NET to make the actual comparison, instead of just executing these instructions in place?
I guess you're a bit confused about how .NET works. C doesn't ask .NET to execute code.
First, your lambda is turned into a compiler-generated class instance (because you're closing over the comparer variable), and then a delegate to a method of this class is used. And it's an instance method since your lambda is a closure.
A delegate is similar to a function pointer. So, like you say, it points to executable code. Whether this code is generated from a C source or a .NET source is irrelevant at this point.
It's in the interop case when this starts to matter. P/Invoke won't pass your delegate as-is as a function pointer to C code. It will pass a function pointer to a thunk which calls the delegate. Visual Studio will display this as a [Native to Managed Transition] stack frame. This is needed for different reasons such as marshaling or passing additional parameters (like the instance of the class backing your lambda for instance).
As to the performance considerations of this, here's what MSDN says, quite obviously:
Thunking. Regardless of the interoperability technique used, special transition sequences, which are known as thunks, are required each time a managed function calls an native function, and vice-versa. Because thunking contributes to the overall time that it takes to interoperate between managed code and native code, the accumulation of these transitions can negatively affect performance.
So, if your code requires a lot of transitions between managed and native code, you should get better performance by doing your comparisons on the C side if possible, so that you avoid the transitions.

marshalling C and C#

I have the following marshalling code in my project. I have few questions on this.
[DllImport=(Core.dll, SetLastError=true, EntryPoint="CoreCreate", CharSet="CharSet.Ansi", CallingConvention="CallingConvention.Cdecl")]
internal static extern uint CoreCreate(ref IntPtr core);
Why 'internal static extern' is required? Is this compulsory? Why this is used?
What is SetLastError?
[StructLayout(LayoutKind.Sequential,CharSet=CharSet.Ansi)]
internal struct Channel
{
internal byte LogicalChannel;
}
Why LayoutKind.Sequential?

Why 'internal static extern' is required?
The internal modifier just set visibility of your method. It's not required to be internal so you can declare the method private or public as you need and as you would do with any other standard method.
The static modifier is required because it's not an instance method and that method doesn't know any class (it hasn't a this pointer).
Finally extern is required to inform the compiler that the method isn't implemented here but in another place (and you'll specify where using attributes). Evey extern method must be declared static too (because it's a simple function call without any knowledge about objects).
What is SetLastError?
It indicates that method may change the thread's last-error code value. See the GetLastError() function for details about this. If the called function will change this value then it's a good thing to set SetLastError to true, from MSDN:
The runtime marshaler calls GetLastError and caches the value returned to prevent it from being overwritten by other API calls. You can retrieve the error code by calling GetLastWin32Error.
In short it saves the value returned by GetLastError() to an internal cache so any other call to system API (even internal to others framework functions) won't overwrite that value.
Why LayoutKind.Sequential?
Class layout in .NET isn't required to be sequential in memory (sequential = if A is declared before B then memory layout has A before B). This is not true in C where the declaration order matters (declaration is used by the compiler to understand the layout, in memory, of raw data). If you have to interop with C functions then you have to be sure about the layout of the data you pass them. This is how LayoutKind.Sequential works: it instructs the compiler to respect the declaration sequential order for data in the struct. This is not the only option to interop with unmanaged world, you can even explicitly set the offset (from structure beginning) of each field (see LayoutKind.Explicit).

This is not an answer, just a few comments:
"internal static" is one thing and "extern" is another thing needed when calling external dll's.
SetLastError or GetLastError is methods we used a lot in the "old" days to get error messages from windows about the latest handling.
LayoutKind.Sequential is a way to inform the compiler to layout the struct in a specified way - you may need to do this if marchalling to other systems.

C# internal static extern with InternalCall attribute - internal or external?

In another question I asked, a comment arose indicating that the .NET framework's Array.Copy method uses unmanaged code. I went digging with Reflector and found the signature one of the Array.Copy method overloads is defined as so:
[MethodImpl(MethodImplOptions.InternalCall), ReliabilityContract(Consistency.MayCorruptInstance, Cer.MayFail)]
internal static extern void Copy(Array sourceArray, int sourceIndex, Array destinationArray, int destinationIndex, int length, bool reliable);
After looking at this, I'm slightly confused. The source of my confusion is the extern modifier which means (MSDN link):
The extern modifier is used to declare
a method that is implemented
externally.
However, the method declaration is also decorated with a MethodImplOptions.InternalCall attribute, which indicates (MSDN link):
Specifies an internal call. An
internal call is a call to a method
that is implemented within the common
language runtime itself.
Can anyone explain this seemingly apparent contradiction?

I would have just commented on leppie's post, but it was getting a bit long.
I'm currently working on an experimental CLI implementation. There are many cases where a publicly exposed method (or property) can't be implemented without knowledge of how the virtual machine is implemented internally. One example is OffsetToStringData, which requires knowledge of how the memory manager allocates strings.
For cases like this, where there is no C# code to express the method, you can treat each call to the method in a special way internal to the JIT process. As an example here, replacing the call byte code with a ldc.i4 (load constant integer) before passing it to the native code generator. The InternalCall flag means "The body of this method is treated in a special way by the runtime itself." There may or may not be an actual implementation - in several cases in my code the call is treated as an intrinsic by the JIT.
There are other cases where the JIT may have special information available that allows heavy optimization of a method. One example is the Math methods, where even though these can be implemented in C#, specifying InternalCall to make them effectively intrinsics has significant performance benefits.
In C#, a method has to have a body unless it is abstract or extern. The extern means a general "You can call this method from C# code, but the body of it is actually defined elsewhere.". When the JIT reaches a call to an extern method, it looks up where to find the body and behaves in different ways per the result.
The DllImport attribute instructs the JIT to make a P/Invoke stub to call a native code implementation.
The InternalCall flag instructs the JIT to treat the call in a self-defined way.
(There are some others, but I don't have examples off the top of my head for their use.)

InternalCall means provided by the framework.
extern says you are not providing code.
extern can be used in 2 general situations, like above, or with p/invoke.
With p/invoke, you simply tell the method where to get the implementation.