Develop Reference

Passing A Byte[] Segment From C# To C++ DLL Without Creating a Copy

I need to slice a byte[] and pass the sliced section to a C# DLL for processing. I am avoiding Array.Copy, because I am trying not to copy anything to hinder performance. I have been made aware of the ArraySegment class as well as Span and Memory. The confusion I am having is how to actually pass these to the DLL, as I am passing an UnmanagedType.LPArray like so:
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate int ProcessVideoFrame_t([MarshalAs(UnmanagedType.LPArray)] byte[] bytes, uint len);
Is there any way to get an underlying byte[] from these classes, or somehow pass the segment without making a copy?
My current code is:
byte[] bytes = new byte[packet.Payload.Length - headerSize];
Array.Copy(packet.Payload, headerSize, bytes, 0, bytes.Length);
helper.ProcessVideoFrame(CookieHelperDictionary[packet.Header.Cookie], bytes, (uint)bytes.Length);```

unsafe
{
fixed (byte* p = &packet.Payload[headerSize])
{
helper.ProcessVideoFrame(CookieHelperDictionary[packet.Header.Cookie], (IntPtr)p, (uint)(packet.Payload.Length - headerSize));
}
}
I also changed the function definition to this:
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate int ProcessVideoFrame_t(IntPtr rdh, IntPtr buffer, uint len);

How to marshall byte* from C.dll in C#

I have two functions that I am trying to call from C# that shares similar signatures:
BOOL Read (BYTE Len, BYTE* DataBuf)
BOOL Write (BYTE Len, BYTE* DataBuf)
From the Doc: DataBuf Destination of transmitted data
What shall I use in C# call?
byte[]
myByteArr[0]
P/Invoke Assistant suggested System.IntPtr
Don't have the hardware to test yet, but i am trying to get as many of calls right for when we have.
Thanks.

For the read function you use:
[Out] byte[] buffer
For the write function you use:
[In] byte[] buffer
[In] is the default and can be omitted but it does not hurt to be explicit.
The functions would therefore be:
[DllImport(filename, CallingConvention = CallingConvention.Cdecl)]
static extern bool Read(byte len, [Out] byte[] buffer);
[DllImport(filename, CallingConvention = CallingConvention.Cdecl)]
static extern bool Write(byte len, [In] byte[] buffer);
Obviously you'll need to allocate the array before passing it to the unmanaged functions.
Because byte is blittable then the marshaller, as an optimisation, pins the array and passes the address of the pinned object. This means that no copying is performed and the parameter passing is efficient.

It'll probably be IntPtr obtained from a byte[] array. Older questions should definitely have covered that: How to get IntPtr from byte[] in C#

Determining DLLImport arguments and safely calling an unmanaged C function

As a follow-up to my previous question, I finally got the C dll exported and usable in C#, but I'm stuck trying to figure out the proper argument types and calling method.
I've researched here on SO but there doesn't seem to be a pattern to how variable types are assigned.
I see some people suggest a StringBuilder for uchar*, others a byte[], some references to 'unsafe' code, etc. Can anyone recommend a solution based on this specific use-case?
Also note the exception generated as the code stands now, right after the call to the C function.
C function import:
[DllImport("LZFuncs.dll")]
internal static extern long LZDecomp(ref IntPtr outputBuffer, byte[] compressedBuffer, UInt32 compBufferLength); //Originally two uchar*, return is size of uncompressed data.
C function signature:
long LZDecomp(unsigned char *OutputBuffer, unsigned char *CompressedBuffer, unsigned long CompBufferLength)
Used as below:
for (int dataNum = 0; dataNum < _numEntries; dataNum++)
{
br.BaseStream.Position = _dataSizes[dataNum]; //Return to start of data.
if (_compressedFlags[dataNum] == 1)
{
_uncompressedSize = br.ReadInt32();
byte[] compData = br.ReadBytes(_dataSizes[dataNum] - 4);
IntPtr outData = IntPtr.Zero;
LZFuncs.LZDecomp(ref outData, compData, Convert.ToUInt32(compData.Length));
var uncompData = new byte[_uncompressedSize]; //System.ExecutionEngineException was unhandled
Marshal.Copy(outData, uncompData, 0, Convert.ToInt32(_uncompressedSize));
BinaryWriter bw = new BinaryWriter(new FileStream("compData" + dataNum + ".txt", FileMode.CreateNew));
bw.Write(uncompData);
bw.Close();
}
else
{
BinaryWriter bw = new BinaryWriter(new FileStream("uncompData" + dataNum + ".txt", FileMode.CreateNew));
bw.Write(br.ReadBytes(_dataSizes[dataNum]));
bw.Close();
}
}
I assume the C code is clobbering the memory pretty severely if it's breaking the C# caller with a CLR exception like that, but due to how the C code is written, there's absolutely no way to modify it without breaking the functionality, it's effectively a black box. (Written in assembly, for the most part.)
For reference, just a few questions I've read over in an effort to solve this myself:
How do I return a byte array from C++ to C#
Correct way to marshall uchar[] from native dll to byte[] in c#
There have been others but those are the most recent.

OK, that's not too hard to work with. The two buffer parameters are byte arrays. You should declare them as byte[]. The calling convention is Cdecl. Remember that C++ long is only 32 bits wide on Windows, so use C# int rather than C# long since the latter is 64 bits wide.
Declare the function like this:
[DllImport("LZFuncs.dll", CallingConvention = CallingConvention.Cdecl)]
internal static extern int LZDecomp(
[Out] byte[] outputBuffer,
[In] byte[] compressedBuffer,
uint compBufferLength
);
You are decompressing compressedBuffer into outputBuffer. You'll need to know how large outputBuffer needs to be (the code in the question shows that you already handle this) and allocate a sufficiently large array. Beyond that I think it's obvious how to call this.
The calling code will this look like this:
_uncompressedSize = br.ReadInt32();
byte[] compData = br.ReadBytes(_dataSizes[dataNum] - 4);
byte[] outData = new byte[_uncompressedSize];
int len = LZFuncs.LZDecomp(outData, compData, (uint)compData.Length);

This is an old question but a real issue and can lead to serious security issues so I thought I give it my 2 cents
Whenever I use [DllImport] I always add the location you consider safe, one option is to specify is for windows DLL's
[DefaultDllImportSearchPaths(DllImportSearchPath.SafeDirectories)]
However, have a look at your options to have it match your needs, you might load a private DLL that is located elsewhere.

Directly reading large binary file in C# w/out copying

I am looking for the most efficient/direct way to do this simple C/C++ operation:
void ReadData(FILE *f, uint16 *buf, int startsamp, int nsamps)
{
fseek(f, startsamp*sizeof(uint16), SEEK_SET);
fread(buf, sizeof(uint16), nsamps, f);
}
in C#/.NET. (I'm ignoring return values for clarity - production code would check them.) Specifically, I need to read in many (potentially 10's to 100's of millions) 2-byte (16-bit) "ushort" integer data samples (fixed format, no parsing required) stored in binary in a disk file. The nice thing about the C way is that it reads the samples directly into the "uint16 *" buffer with no CPU involvement, and no copying. Yes, it is potentially "unsafe", as it uses void * pointers to buffers of unknown size, but it seems like there should be a "safe" .NET alternative.
What is the best way to accomplish this in C#? I have looked around, and come across a few hints ("unions" using FieldOffset, "unsafe" code using pointers, Marshalling), but none seem to quite work for this situation, w/out using some sort of copying/conversion. I'd like to avoid BinaryReader.ReadUInt16(), since that is very slow and CPU intensive. On my machine there is about a 25x difference in speed between a for() loop with ReadUInt16(), and reading the bytes directly into a byte[] array with a single Read(). That ratio could be even higher with non-blocking I/O (overlapping "useful" processing while waiting for the disk I/O).
Ideally, I would want to simply "disguise" a ushort[] array as a byte[] array so I could fill it directly with Read(), or somehow have Read() fill the ushort[] array directly:
// DOES NOT WORK!!
public void GetData(FileStream f, ushort [] buf, int startsamp, int nsamps)
{
f.Position = startsamp*sizeof(ushort);
f.Read(buf, 0, nsamps);
}
But there is no Read() method that takes a ushort[] array, only a byte[] array.
Can this be done directly in C#, or do I need to use unmanaged code, or a third-party library, or must I resort to CPU-intensive sample-by-sample conversion? Although "safe" is preferred, I am fine with using "unsafe" code, or some trick with Marshal, I just have not figured it out yet.
Thanks for any guidance!
[UPDATE]
I wanted to add some code as suggested by dtb, as there seem to be precious few examples of ReadArray around. This is a very simple one, w/no error checking shown.
public void ReadMap(string fname, short [] data, int startsamp, int nsamps)
{
var mmf = MemoryMappedFile.CreateFromFile(fname);
var mmacc = mmf.CreateViewAccessor();
mmacc.ReadArray(startsamp*sizeof(short), data, 0, nsamps);
}
Data is safely dumped into your passed array. You can also specify a type for more complex types. It seems able to infer simple types on its own, but with the type specifier, it would look like this:
mmacc.ReadArray<short>(startsamp*sizeof(short), data, 0, nsamps);
[UPATE2]
I wanted to add the code as suggested by Ben's winning answer, in "bare bones" form, similar to above, for comparison. This code was compiled and tested, and works, and is FAST. I used the SafeFileHandle type directly in the DllImport (instead of the more usual IntPtr) to simplify things.
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool ReadFile(SafeFileHandle handle, IntPtr buffer, uint numBytesToRead, out uint numBytesRead, IntPtr overlapped);
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool SetFilePointerEx(SafeFileHandle hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);
unsafe void ReadPINV(FileStream f, short[] buffer, int startsamp, int nsamps)
{
long unused; uint BytesRead;
SafeFileHandle nativeHandle = f.SafeFileHandle; // clears Position property
SetFilePointerEx(nativeHandle, startsamp*sizeof(short), out unused, 0);
fixed(short* pFirst = &buffer[0])
ReadFile(nativeHandle, (IntPtr)pFirst, (uint)nsamps*sizeof(short), out BytesRead, IntPtr.Zero);
}

You can use a MemoryMappedFile. After you have memory-mapped the file, you can create a view (i.e. a MemoryMappedViewAccessor) which provides a ReadArray<T> method. This method can read structs from the file without marshalling, and it works with primitive types lie ushort.

dtb's answer is an even better way (actually, it has to copy the data as well, no gain there), but I just wanted to point out that to extract ushort values from a byte array you should be using BitConverter not BinaryReader
EDIT: example code for p/invoking ReadFile:
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool ReadFile(IntPtr handle, IntPtr buffer, uint numBytesToRead, out uint numBytesRead, IntPtr overlapped);
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool SetFilePointerEx(IntPtr hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);
unsafe bool read(FileStream fs, ushort[] buffer, int offset, int count)
{
if (null == fs) throw new ArgumentNullException();
if (null == buffer) throw new ArgumentNullException();
if (offset < 0 || count < 0 || offset + count > buffer.Length) throw new ArgumentException();
uint bytesToRead = 2 * count;
if (bytesToRead < count) throw new ArgumentException(); // detect integer overflow
long offset = fs.Position;
SafeFileHandle nativeHandle = fs.SafeFileHandle; // clears Position property
try {
long unused;
if (!SetFilePositionEx(nativeHandle, offset, out unused, 0);
fixed (ushort* pFirst = &buffer[offset])
if (!ReadFile(nativeHandle, new IntPtr(pFirst), bytesToRead, out bytesToRead, IntPtr.Zero)
return false;
if (bytesToRead < 2 * count)
return false;
offset += bytesToRead;
return true;
}
finally {
fs.Position = offset; // restore Position property
}
}

I might be a bit late to the game here... but the fastest method I found was using a combination of the previous answers.
If i do the following:
MemoryMappedFile mmf = MemoryMappedFile.CreateFromFile(somePath);
Stream io = mmf.CreateViewStream();
int count;
byte[] byteBuffer = new byte[1024 << 2];
ushort[] dataBuffer = new ushort[buffer.Length >> 1];
while((count = io.Read(byteBuffer, 0, byteBuffer.Length)) > 0)
Buffer.BlockCopy(buffer, 0, dataBuffer, 0, count);
This was ~2x faster than the accepted answer.
For me, the unsafe method was the same as the Buffer.BlockCopy without the MemoryMappedFile. The MemoryMappedFile cut down on a bit of time.

How to get IntPtr from byte[] in C#

I want to pass a byte[] to a method takes a IntPtr Parameter in C#, is that possible and how?

Another way,
GCHandle pinnedArray = GCHandle.Alloc(byteArray, GCHandleType.Pinned);
IntPtr pointer = pinnedArray.AddrOfPinnedObject();
// Do your stuff...
pinnedArray.Free();

The following should work but must be used within an unsafe context:
byte[] buffer = new byte[255];
fixed (byte* p = buffer)
{
IntPtr ptr = (IntPtr)p;
// Do your stuff here
}
Beware: you have to use the pointer within the fixed block. The GC can move the object once you are no longer within the fixed block.

Not sure about getting an IntPtr to an array, but you can copy the data for use with unmanaged code by using Mashal.Copy:
IntPtr unmanagedPointer = Marshal.AllocHGlobal(bytes.Length);
Marshal.Copy(bytes, 0, unmanagedPointer, bytes.Length);
// Call unmanaged code
Marshal.FreeHGlobal(unmanagedPointer);
Alternatively you could declare a struct with one property and then use Marshal.PtrToStructure, but that would still require allocating unmanaged memory.
Edit: Also, as Tyalis pointed out, you can also use fixed if unsafe code is an option for you

You could use Marshal.UnsafeAddrOfPinnedArrayElement to get a memory pointer to the array (or to a specific element in the array). Keep in mind that the array must be pinned first as per the API documentation:
The array must be pinned using a GCHandle before it is passed to this method. For maximum performance, this method does not validate the array passed to it; this can result in unexpected behavior.

Here's a twist on #user65157's answer (+1 for that, BTW):
I created an IDisposable wrapper for the pinned object:
class AutoPinner : IDisposable
{
GCHandle _pinnedArray;
public AutoPinner(Object obj)
{
_pinnedArray = GCHandle.Alloc(obj, GCHandleType.Pinned);
}
public static implicit operator IntPtr(AutoPinner ap)
{
return ap._pinnedArray.AddrOfPinnedObject();
}
public void Dispose()
{
_pinnedArray.Free();
}
}
then use it like thusly:
using (AutoPinner ap = new AutoPinner(MyManagedObject))
{
UnmanagedIntPtr = ap; // Use the operator to retrieve the IntPtr
//do your stuff
}
I found this to be a nice way of not forgetting to call Free() :)

Marshal.Copy works but is rather slow. Faster is to copy the bytes in a for loop. Even faster is to cast the byte array to a ulong array, copy as much ulong as fits in the byte array, then copy the possible remaining 7 bytes (the trail that is not 8 bytes aligned). Fastest is to pin the byte array in a fixed statement as proposed above in Tyalis' answer.

In some cases you can use an Int32 type (or Int64) in case of the IntPtr. If you can, another useful class is BitConverter. For what you want you could use BitConverter.ToInt32 for example.