I want to convert a byte* to a byte[], but I also want to have a reusable function to do this:
public unsafe static T[] Create<T>(T* ptr, int length)
{
T[] array = new T[length];
for (int i = 0; i < length; i++)
array[i] = ptr[i];
return array;
}
Unfortunately I get a compiler error because T might be a ".NET managed type" and we can't have pointers to those. Even more frustrating is that there is no generic type constraint which can restrict T to "unmanaged types". Is there a built-in .NET function to do this? Any ideas?
The method that could match what you are trying to do is Marshal.Copy, but it does not take the appropriate parameters to make a generic method.
Although there it is not possible to write a generic method with generic constraints that could describe what is possible, not every type can be allowed to be copied using an "unsafe" way. There are some exceptions; classes are one of these.
Here is a sample code:
public unsafe static T[] Create<T>(void* source, int length)
{
var type = typeof(T);
var sizeInBytes = Marshal.SizeOf(typeof(T));
T[] output = new T[length];
if (type.IsPrimitive)
{
// Make sure the array won't be moved around by the GC
var handle = GCHandle.Alloc(output, GCHandleType.Pinned);
var destination = (byte*)handle.AddrOfPinnedObject().ToPointer();
var byteLength = length * sizeInBytes;
// There are faster ways to do this, particularly by using wider types or by
// handling special lengths.
for (int i = 0; i < byteLength; i++)
destination[i] = ((byte*)source)[i];
handle.Free();
}
else if (type.IsValueType)
{
if (!type.IsLayoutSequential && !type.IsExplicitLayout)
{
throw new InvalidOperationException(string.Format("{0} does not define a StructLayout attribute", type));
}
IntPtr sourcePtr = new IntPtr(source);
for (int i = 0; i < length; i++)
{
IntPtr p = new IntPtr((byte*)source + i * sizeInBytes);
output[i] = (T)System.Runtime.InteropServices.Marshal.PtrToStructure(p, typeof(T));
}
}
else
{
throw new InvalidOperationException(string.Format("{0} is not supported", type));
}
return output;
}
unsafe static void Main(string[] args)
{
var arrayDouble = Enumerable.Range(1, 1024)
.Select(i => (double)i)
.ToArray();
fixed (double* p = arrayDouble)
{
var array2 = Create<double>(p, arrayDouble.Length);
Assert.AreEqual(arrayDouble, array2);
}
var arrayPoint = Enumerable.Range(1, 1024)
.Select(i => new Point(i, i * 2 + 1))
.ToArray();
fixed (Point* p = arrayPoint)
{
var array2 = Create<Point>(p, arrayPoint.Length);
Assert.AreEqual(arrayPoint, array2);
}
}
The method can be generic, but it cannot take a pointer of a generic type. This is not an issue since pointers covariance is helping, but this has the unfortunate effect of preventing an implicit resolution of the generic argument type. You then have to specify MakeArray explicitly.
I've added a special case for the structures, where it is best to have types that specify a struct layout. This might not be an issue in your case, but if the pointer data is coming from native C or C++ code, specifying a layout kind is important (The CLR might choose to reorder fields to have a better memory alignment).
But if the pointer is coming exclusively from data generated by managed code, then you can remove the check.
Also, if the performance is an issue, there are better algorithms to copy the data than doing it byte by byte. (See the countless implementations of memcpy for reference)
Seems that the question becomes: How to specify a generic Type to be a simple type.
unsafe void Foo<T>() : where T : struct
{
T* p;
}
Gives the error:
Cannot take the address of, get the size of, or declare a pointer to a managed type ('T')
How about this?
static unsafe T[] MakeArray<T>(void* t, int length, int tSizeInBytes) where T:struct
{
T[] result = new T[length];
for (int i = 0; i < length; i++)
{
IntPtr p = new IntPtr((byte*)t + (i * tSizeInBytes));
result[i] = (T)System.Runtime.InteropServices.Marshal.PtrToStructure(p, typeof(T));
}
return result;
}
We can't use sizeof(T) here, but the caller can do something like
byte[] b = MakeArray<byte>(pBytes, lenBytes, sizeof(byte));
I have no idea whatsoever if the following would work, but it might (at least it compiles :):
public unsafe static T[] Create<T>(void* ptr, int length) where T : struct
{
T[] array = new T[length];
for (int i = 0; i < length; i++)
{
array[i] = (T)Marshal.PtrToStructure(new IntPtr(ptr), typeof(T));
}
return array;
}
The key is to use Marshal.PtrToStructure to convert to the correct type.
As of C# 7.3 this is possible as follows:
public unsafe static T[] Create<T>(T* ptr, int length) where T: unmanaged
{
T[] array = new T[length];
for (int i = 0; i < length; i++)
array[i] = ptr[i];
return array;
}
Related
I have a generic class
class UnmanagedArray<T> where T: struct
{
IntPtr _backing; // unmanaged memory
// some constructors and others methods
// very slow but I don't care
public T this[int index]
{
get
{
int elementSize = Marshal.SizeOf(default(T));
byte* ptr = ((byte*)tab) + index * elementSize;
T result = default(T);
var handle = GCHandle.Alloc(result, GCHandleType.Pinned);
byte* result_ptr = (byte*) handle.AddrOfPinnedObject();
for(int k = 0; k < elementSize; ++k)
{
result_ptr[k] = ptr[k];
}
handle.Free();
return result;
}
}
}
Now I'd like to modify elements in my array.
Something like
myArray[i].prop = 12;
I can't do it because
cannot modify the return value of Array.this[int] because it is
not a variable.
I totally understand the error, since T is a ValueType, therefore immutable (returned by copy).
Is there any way to make a ref return in my indexer property?
PS : T cannot be a reference type. It has to be a ValueType.
I have float array inside C++ function.
C++ function
void bleplugin_GetGolfResult(float* result)
{
float *array = new float[20];
for(int i=0; i < 20; i++)
array[i]= 25;
result = array;
//DEBUG PRINTING1
for(int i=0; i < 20; i++)
cout << result[i] << endl;//Here is correct
return;
}
Inside C#
[DllImport ("__Internal")]
private static unsafe extern void bleplugin_GetGolfResult (float* result);
public static unsafe float[] result = new float[20];
public unsafe static void GetGolfREsult(){
fixed (float* ptr_result = result) //or equivalently "... = &f2[0]" address of f2[0]
{
bleplugin_GetGolfResult( ptr_result );
//DEBUG PRINTING2
for(int i = 0; i < 20; i++)
Debug.Log("result data " + ptr_result[i]);
}
return;
}
I called GetGolfREsult() from another function to get result.
//DEBUG PRINTING1 has correct output.
But //DEBUG PRINTING2 produced 0 only.
What could be wrong?
As UnholySheep and nvoigt already stated,
result = array;
overrides the address of the passed pointer, making you lose the reference to the calling function.
Directly writing to your parameter should solve this.
result[i] = 25;
Further you dont actually have to use pointers in c#.
You can actually do the following:
Declare your Import like this:
private static extern void bleplugin_GetGolfResult (float arr[]);
Then you can call it like this:
float arr = new float[20];
bleplugin_GetGolfResult(arr);
This line in your C++ code:
float *array = new float[20];
creates a new array, which you operate on in C++. Then control returns to C#, who has it's own array and that's still unchanged. Why don't you write to the array you got?
The problem is that you use the assignment operator on the parameter result which prevents the data from being transferred to the C# array on return.
Using the following C++ example:
void z(int * x)
{
x = new int(4);
}
int main()
{
int * x = new int(-2);
z(x);
cout<<*x<<endl;
}
The output for this is -2 not 4 because you use the assignment operator on the parameter.
Hello I'm wrapping C++ library with C#. Next function in C++:
SCREENCAPTUREDLL_API wchar_t** getAudioDeviceList(int* listSize) {
static std::vector<wchar_t*> descriptionList;
AudioCaptureList::getInstance().Update();
AudioCaptureList::getInstance().getList(&descriptionList);
*listSize = descriptionList.size();
return &descriptionList[0];
}
Wrapping with next C# code:
[DllImport(screenCaptureDLLPath, CallingConvention = callConversion)]
private static extern IntPtr getAudioDeviceList(ref int arrayCount);
public static string[] GetAudioDeviceList()
{
IntPtr outputStr;
int length = 0;
outputStr = getAudioDeviceList(ref length);
string[] resultArray = new string[length];
for (int j = 0; j < length; j++)
{
resultArray[j] = Marshal.PtrToStringUni(Marshal.ReadIntPtr(outputStr, 4 * j));
}
return resultArray;
}
That works perfect, exactly as I expected, but I was about to change the way I returning value from function itself to variable by reference, so I changing my code to:
C++
SCREENCAPTUREDLL_API void getAudioDeviceList(wchar_t** list, int* listSize) {
static std::vector<wchar_t*> descriptionList;
AudioCaptureList::getInstance().Update();
AudioCaptureList::getInstance().getList(&descriptionList);
*listSize = descriptionList.size();
list = &descriptionList[0];
}
C#
[DllImport(screenCaptureDLLPath, CallingConvention = callConversion)]
private static extern void getAudioDeviceList(out IntPtr listRef, ref int arrayCount);
public static string[] GetAudioDeviceList()
{
IntPtr outputStr;
int length = 0;
getAudioDeviceList(out outputStr, ref length);
string[] resultArray = new string[length];
for (int j = 0; j < length; j++)
{
resultArray[j] = Marshal.PtrToStringUni(Marshal.ReadIntPtr(outputStr, 4 * j));
}
return resultArray;
}
But I got error, returned memory address is zero. What is the problem here? Please help me understood what cause the problem and how to fix that, thanks!
Why doesn't Pinvoke work? Because you are trying to interpret a pointer to a string as a pointer to a set of strings. But there is nothing wrong with PInvoke - it happens because there is actually a problem with new function signature and its internal code.
See:
SCREENCAPTUREDLL_API void getAudioDeviceList(wchar_t** listRef, int* listSize);
can't provide the same data like
DLL_API wchar_t** getAudioDeviceList(int* listSize)
Because original definition basically returned pointer to a set of pointers to strings(C style strings, I mean), while wchar_t** listRef can only allow to return a single pointer to a string.
SCREENCAPTUREDLL_API void getAudioDeviceList(wchar_t** listRef, int* listSize)
{
...
*listRef = "string";
I don't know what is going inside new version of the function(you didn't show the code), but listRef = &descriptionList[0]; will compile though won't do anything, and even if *listRef = &descriptionList[0]; somehow compiles it won't contain what you want.
So function signature should containt triple pointer to allow return of a set of strings.
SCREENCAPTUREDLL_API void getAudioDeviceList(wchar_t*** listRef, int* listSize)
{
...
*listRef = &descriptionList[0];
}
Then your PInvoke would work correctly because it will have the same pointer to a set of string pointers.
The idea: Being able to take the bytes of any struct, send those bytes across a TcpClient (or through my Client wrapper), then have the receiving client load those bytes and use pointers to "paint" them onto a new struct.
The problem: It reads the bytes into the buffer perfectly; it reads the array of bytes on the other end perfectly. The "paint" operation, however, fails miserably. I write a new Vector3(1F, 2F, 3F); I read a Vector3(0F, 0F, 0F)...Obviously, not ideal.
Unfortunately, I don't see the bug - If it works one way, it should work the reverse - And the values are being filled in.
The write/read functions are as follows:
public static unsafe void Write<T>(Client client, T value) where T : struct
{
int n = System.Runtime.InteropServices.Marshal.SizeOf(value);
byte[] buffer = new byte[n];
{
var handle = System.Runtime.InteropServices.GCHandle.Alloc(value, System.Runtime.InteropServices.GCHandleType.Pinned);
void* ptr = handle.AddrOfPinnedObject().ToPointer();
byte* bptr = (byte*)ptr;
for (int t = 0; t < n; ++t)
{
buffer[t] = *(bptr + t);
}
handle.Free();
}
client.Writer.Write(buffer);
}
Line Break
public static unsafe T Read<T>(Client client) where T : struct
{
T r = new T();
int n = System.Runtime.InteropServices.Marshal.SizeOf(r);
{
byte[] buffer = client.Reader.ReadBytes(n);
var handle = System.Runtime.InteropServices.GCHandle.Alloc(r, System.Runtime.InteropServices.GCHandleType.Pinned);
void* ptr = handle.AddrOfPinnedObject().ToPointer();
byte* bptr = (byte*)ptr;
for (int t = 0; t < n; ++t)
{
*(bptr + t) = buffer[t];
}
handle.Free();
}
return r;
}
Help, please, thanks.
Edit:
Well, one major problem is that I'm getting a handle to a temporary copy created when I passed in the struct value.
Edit2:
Changing "T r = new T();" to "object r = new T();" and "return r" to "return (T)r" boxes and unboxes the struct and, in the meantime, makes it a reference, so the pointer actually points to it.
However, it is slow. I'm getting 13,500 - 14,500 write/reads per second.
Edit3:
OTOH, serializing/deserializing the Vector3 through a BinaryFormatter gets about 750 writes/reads per second. So a Lot faster than what I was using. :)
Edit4:
Sending the floats individually got 8,400 RW/second. Suddenly, I feel much better about this. :)
Edit5:
Tested GCHandle allocation pinning and freeing; 28,000,000 ops per second (Compared to about 1,000,000,000 Int32/int add+assign/second. So compared to integers, it's 35 times slower. However, that's still comparatively fast enough). Note that you don't appear to be able to pin classes, even if GCHandle does auto-boxed structs fine (GCHandle accepts values of type "object").
Now, if the C# guys update constraints to the point where the pointer allocation recognizes that "T" is a struct, I could just assign directly to a pointer, which is...Yep, incredibly fast.
Next up: Probably testing write/read using separate threads. :) See how much the GCHandle actually affects the send/receive delay.
As it turns out:
Edit6:
double start = Timer.Elapsed.TotalSeconds;
for (t = 0; t < count; ++t)
{
Vector3 from = new Vector3(1F, 2F, 3F);
// Vector3* ptr = &test;
// Vector3* ptr2 = &from;
int n = sizeof(Vector3);
if (n / 4 * 4 != n)
{
// This gets 9,000,000 ops/second;
byte* bptr1 = (byte*)&test;
byte* bptr2 = (byte*)&from;
// int n = 12;
for (int t2 = 0; t2 < n; ++t2)
{
*(bptr1 + t2) = *(bptr2 + t2);
}
}
else
{
// This speedup gets 24,000,000 ops/second.
int n2 = n / 4;
int* iptr1 = (int*)&test;
int* iptr2 = (int*)&from;
// int n = 12;
for (int t2 = 0; t2 < n2; ++t2)
{
*(iptr1 + t2) = *(iptr2 + t2);
}
}
}
So, overall, I don't think the GCHandle is really slowing things down. (Those who are thinking this is a slow way of assigning one Vector3 to another, remember that the purpose is to serialize structs into a byte[] buffer to send over a network. And, while that's not what we're doing here, it would be rather easy to do with the first method).
Edit7:
The following got 6,900,000 ops/second:
for (t = 0; t < count; ++t)
{
Vector3 from = new Vector3(1F, 2F, 3F);
int n = sizeof(Vector3);
byte* bptr2 = (byte*)&from;
byte[] buffer = new byte[n];
for (int t2 = 0; t2 < n; ++t2)
{
buffer[t2] = *(bptr2 + t2);
}
}
...Help! I've got IntruigingPuzzlitus! :D
Is there a way of mapping data collected on a stream or array to a data structure or vice-versa?
In C++ this would simply be a matter of casting a pointer to the stream as a data type I want to use (or vice-versa for the reverse)
eg: in C++
Mystruct * pMyStrct = (Mystruct*)&SomeDataStream;
pMyStrct->Item1 = 25;
int iReadData = pMyStrct->Item2;
obviously the C++ way is pretty unsafe unless you are sure of the quality of the stream data when reading incoming data, but for outgoing data is super quick and easy.
Most people use .NET serialization (there is faster binary and slower XML formatter, they both depend on reflection and are version tolerant to certain degree)
However, if you want the fastest (unsafe) way - why not:
Writing:
YourStruct o = new YourStruct();
byte[] buffer = new byte[Marshal.SizeOf(typeof(YourStruct))];
GCHandle handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
Marshal.StructureToPtr(o, handle.AddrOfPinnedObject(), false);
handle.Free();
Reading:
handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
o = (YourStruct)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(YourStruct));
handle.Free();
In case lubos hasko's answer was not unsafe enough, there is also the really unsafe way, using
pointers in C#. Here's some tips and pitfalls I've run into:
using System;
using System.Runtime.InteropServices;
using System.IO;
using System.Diagnostics;
// Use LayoutKind.Sequential to prevent the CLR from reordering your fields.
[StructLayout(LayoutKind.Sequential)]
unsafe struct MeshDesc
{
public byte NameLen;
// Here fixed means store the array by value, like in C,
// though C# exposes access to Name as a char*.
// fixed also requires 'unsafe' on the struct definition.
public fixed char Name[16];
// You can include other structs like in C as well.
public Matrix Transform;
public uint VertexCount;
// But not both, you can't store an array of structs.
//public fixed Vector Vertices[512];
}
[StructLayout(LayoutKind.Sequential)]
unsafe struct Matrix
{
public fixed float M[16];
}
// This is how you do unions
[StructLayout(LayoutKind.Explicit)]
unsafe struct Vector
{
[FieldOffset(0)]
public fixed float Items[16];
[FieldOffset(0)]
public float X;
[FieldOffset(4)]
public float Y;
[FieldOffset(8)]
public float Z;
}
class Program
{
unsafe static void Main(string[] args)
{
var mesh = new MeshDesc();
var buffer = new byte[Marshal.SizeOf(mesh)];
// Set where NameLen will be read from.
buffer[0] = 12;
// Use Buffer.BlockCopy to raw copy data across arrays of primitives.
// Note we copy to offset 2 here: char's have alignment of 2, so there is
// a padding byte after NameLen: just like in C.
Buffer.BlockCopy("Hello!".ToCharArray(), 0, buffer, 2, 12);
// Copy data to struct
Read(buffer, out mesh);
// Print the Name we wrote above:
var name = new char[mesh.NameLen];
// Use Marsal.Copy to copy between arrays and pointers to arrays.
unsafe { Marshal.Copy((IntPtr)mesh.Name, name, 0, mesh.NameLen); }
// Note you can also use the String.String(char*) overloads
Console.WriteLine("Name: " + new string(name));
// If Erik Myers likes it...
mesh.VertexCount = 4711;
// Copy data from struct:
// MeshDesc is a struct, and is on the stack, so it's
// memory is effectively pinned by the stack pointer.
// This means '&' is sufficient to get a pointer.
Write(&mesh, buffer);
// Watch for alignment again, and note you have endianess to worry about...
int vc = buffer[100] | (buffer[101] << 8) | (buffer[102] << 16) | (buffer[103] << 24);
Console.WriteLine("VertexCount = " + vc);
}
unsafe static void Write(MeshDesc* pMesh, byte[] buffer)
{
// But byte[] is on the heap, and therefore needs
// to be flagged as pinned so the GC won't try to move it
// from under you - this can be done most efficiently with
// 'fixed', but can also be done with GCHandleType.Pinned.
fixed (byte* pBuffer = buffer)
*(MeshDesc*)pBuffer = *pMesh;
}
unsafe static void Read(byte[] buffer, out MeshDesc mesh)
{
fixed (byte* pBuffer = buffer)
mesh = *(MeshDesc*)pBuffer;
}
}
if its .net on both sides:
think you should use binary serialization and send the byte[] result.
trusting your struct to be fully blittable can be trouble.
you will pay in some overhead (both cpu and network) but will be safe.
If you need to populate each member variable by hand you can generalize it a bit as far as the primitives are concerned by using FormatterServices to retrieve in order the list of variable types associated with an object. I've had to do this in a project where I had a lot of different message types coming off the stream and I definitely didn't want to write the serializer/deserializer for each message.
Here's the code I used to generalize the deserialization from a byte[].
public virtual bool SetMessageBytes(byte[] message)
{
MemberInfo[] members = FormatterServices.GetSerializableMembers(this.GetType());
object[] values = FormatterServices.GetObjectData(this, members);
int j = 0;
for (int i = 0; i < members.Length; i++)
{
string[] var = members[i].ToString().Split(new char[] { ' ' });
switch (var[0])
{
case "UInt32":
values[i] = (UInt32)((message[j] << 24) + (message[j + 1] << 16) + (message[j + 2] << 8) + message[j + 3]);
j += 4;
break;
case "UInt16":
values[i] = (UInt16)((message[j] << 8) + message[j + 1]);
j += 2;
break;
case "Byte":
values[i] = (byte)message[j++];
break;
case "UInt32[]":
if (values[i] != null)
{
int len = ((UInt32[])values[i]).Length;
byte[] b = new byte[len * 4];
Array.Copy(message, j, b, 0, len * 4);
Array.Copy(Utilities.ByteArrayToUInt32Array(b), (UInt32[])values[i], len);
j += len * 4;
}
break;
case "Byte[]":
if (values[i] != null)
{
int len = ((byte[])values[i]).Length;
Array.Copy(message, j, (byte[])(values[i]), 0, len);
j += len;
}
break;
default:
throw new Exception("ByteExtractable::SetMessageBytes Unsupported Type: " + var[1] + " is of type " + var[0]);
}
}
FormatterServices.PopulateObjectMembers(this, members, values);
return true;
}