Is it possible with flatbuffers in C# to serialize objects to native (unmanaged) memory buffer?
So I want to do these steps:
Allocate a native memory buffer from native memory
Create objects in C# and serialize them into the allocated buffer
Send this memory buffer to C++ for deserialization
I'm thinking either of some custom memory buffer allocator in C#, or of some way of transferring ownership of a memory buffer form C# to C++.
In general I want to avoid copying memory when sending data from C# to C++ and vice versa. I want this memory buffer to be shared between C# and C++.
How do I do that?
No, the current FlatBuffers implementation is hard-coded to write to a regular byte array. You could copy this array to native memory afterwards, or like #pm100 says, pin it.
All serialization in FlatBuffers goes through an abstraction called the ByteBuffer, so if you made an implementation of that for native memory, it could be used directly relatively easily.
Yes, if you use C++/CLI. Basic data types such as bool, 32-bit int, short, etc are same. For other types check it out msclr::interop::marshal_as<>.
Similar post: C++/CLI Converting from System::String^ to std::string
Related
The question is general but I am actually doing this with Mono and not .net, so if there are differences, I am very interested in what they are.
I have a simple data containing class (not struct for other reasons) which should be blitable in the sense that it consist of ints and doubles and smaller structs which consist of doubles. I am sending this to a native dll through DllImport static methods as a reference.
I was under the impression that for a simple object like that, what happens is that it is pinned in managed memory, the address of it, in managed memory, is passed to the native code as a reference/pointer (depending on how the native code is declared, same thing), the native code can read and write it, the function returns and the managed object is unpinned and may now hold changes written by the native code.
Others think the object is instead copied into a block of native memory and then the native code runs on it after which the data is copied back into the object in managed memory. This obviously being less performent and wasteful when the marshaling does not have to convert the data.
I made a test where I note the address of the data sent to native code and I see that it does not change per object. ObjectA gets one address, objectB gets another and each keep their address for as long as I tested.... but while that seems to support my understanding of this, there could still be other explanations for the addresses, so I would be grateful for a concrete explanation, since Mono documents do not mention pinning blittable objects while Microsoft documentation does.
Extra question:
Can there can be a situation with a class (not struct) containing only ushort, int and double where it is not blittable but requires a copy? It was observed with mono on android that changes to the data in native code native would not be visible on managed side (when not using the Out decoration), seemingly indicating that copying, and not pinning, was used.
It is possible that the mono int may be different from the c++ int in the native code, but such data size and alignment issues should not be detectable from the managed side, so how would it "know" to marshal by copy and not pinning? In tests on windows such mismatch just result in garbled data, as expected, so that is likely not the reason for marshal by copy.
What is the best way to copy a string into a raw memory buffer in C#?
Note that I already know how and when to use String and StringBuilder, so don't suggest that ;) - I'm going to need some text processing & rendering code and currently it looks like a memory buffer is both easier to code and more performant, as long as I can get the data into it. (I'm thinking of B-tree editor buffers and memory mapped files, something which doesn't map well into managed C# objects but is easily coded with pointers.)
Things I already considered:
C++/CLI can do the thing, there is PtrToStringChars in vcclr.h which can then be passed to memcpy, but I'm usually preferring only having one assembly and merging the IL from multiple languages is something I like to avoid. Any way to rewrite that function in C#?
System.Runtime.InteropServices.Marshal has functions which copy the string, but only to a newly allocated buffer. Couldn't find any function to copy into an existing buffer.
I could use String.CopyTo and use an array instead of a memory buffer, but then I need to pin that buffer a lot (or keep it pinned all the time) which is going to be bad for GC. (By using a memory buffer in the first place I can allocate it outside the managed heap so it doesn't mess with the GC.)
If there's a way to pin or copy a StringBuilder then that would probably work too. My text usually comes from either a file or a StringBuilder, so if I can already move it into the memory buffer at that point it never needs to go through a String instance. (Note that going from StringBuilder to String doesn't matter for performance because this is optimized to not make a copy if you stop using the StringBuilder afterwards.)
Can I generate IL which pins a String or StringBuilder? Then instead of writing the copy-function in C# I could generate a DynamicMethod by emitting the required IL. Just now thought of this while writing the question, so I might just try to disassembly the C++/CLI way and reproduce the IL.
enable unsafe code(Somewhere in the project options), then use:
unsafe
{
fixed(char* pc = myString)
{
}
}
and then just use low level memory copies.
there's a WinForms-application written in C# using .NET Framework 3.5. This application uses a C++ Dll which is imported using the following declaration:
[DllImport(DllName)]
public static unsafe extern int LoadDBData(String dsn, String userid, String password);
This method imports data from a given ODBC-DSN using a SQL Server database. The call crashes when there is too much data in database. The provider of this extern dll said this happens because the dll is unable to grab more heap size and my application should provide more heap memory.
How could I solve this problem? As far as I know the only possibility to exclude a component from automatic garbage collection is the unsafe keyword which I already used.
Any idea would be appreciated.
Thanks in advance
Martin
This seems like a problem with the vendor's library, rather than your code.
Managed and unmanaged memory should be considered to be completely separate. Managed memory is typically memory allocated on a garbage-collected
heap, while unmanaged memory is anything else: the ANSI C memory pool
allocated through malloc(3), custom memory pools, and
garbage-allocated heaps outside the control of the CLI implementation...
Note that the above quote is from the Mono documentation, but I believe (if I'm not mistaken) the same is true for .NET in general. If the data is being loaded in the DLL's internal data structures, then it should allocate its own memory. If you're providing a buffer which will get filled up with the data, then it will only get filled up with as much data as you've allocated for the buffer (and pinned before marshalling). So where is the data being loaded?
You can't increase the heap size in .NET.
You could create an EXE in c/c++ that your .NET app calls using Process.Start.
Your c/c++ EXE would just call the DLL function and return the result (or if you have more than one function it could take a command line parameter). If you don't want a separate EXE you could try using RunDll32 instead.
I doubt this is specific to .NET, managed memory, garbage collection etc. It's a native DLL so it uses regular, unmanaged memory. Of course, the .NET runtime will also use it's share of memory but a native application using the DLL would do the same.
If you're running in a 32 bit process, the total heap size for .NET and unmanaged code can be limited to 1.5 GB. It's difficult to tell without additional information, but you might have hit that limit.
So one option would be to ask your vendor, whether they have a 64 bit version of the library and switch to a 64 process. In a 64 bit process, memory is almost unlimited (according to today's standard).
Is it possible on windows for unmanaged code (c++ / c) to write to an area in memory that is then accessed by managed .Net code (c#) (separate processes) I have a c program that is writing data to an circular memory buffer and I want to process the buffer with unmanaged code.
If you're looking for a solution where an unmanaged process and managed process can share memory, then you can use the MemoryMappedFile class (introduced in .NET 4.0).
If you're looking to share memory between unmaanged and managed code in the same process, then you can use GCHandle to pin a managed array in memory, and pass it to unmanaged code which can access it.
I think you're looking for Marshaling
Yes,
you should use the Marshall class, especially Marshal.AllocHGlobal..
Yes, take a look at the unsafe and fixed keywords.
what if I had a native C++ function in which, depending on the result of the function, the responsibility of deleting a certain pointer (delete[]) differs between the caller and the function. I would of course check for the return value and act accordingly in C++.
Question is, what if the function was marshalled between C++ and C#, will setting the pointer to null in C# be enough?
No. C# can't do what delete[] in C++ does. You'd have to use a shared memory allocation API, or write a C++ wrapper that handles the cleanup.
No, simply setting a pointer allocated in native code to null will not free the memory. The CLR can only garbage collect memory that it knows about (aka managed memory). It has no idea about native memory and hence can't collect it. Any native memory which has ownership in a managed type must be explicitly freed.
The most common way this is done is via the Alloc and Free functions on the Marshal class
http://msdn.microsoft.com/en-us/library/atxe881w.aspx