I have a managed array provided by a third party library. The type of the array does not directly reflect the data stored in the array, but instead it's the data interpreted as integers.
So int[] data = Lib.GetData(); gives me an integer array, that I would like to cast into an array of DataStructure, that could look like this.
struct DataStructure {
public int Id;
public double Value;
}
Current I use Marshal.Copy (an implementation can been seen here), but it seems a bit excessive to copy the entire thing.
Does something like this exists:
int[] data = Lib.GetData();
DataStructure[] dataStructs = InterpretAs<DataStructure>(data);
where no copying is required, but access to dataStruct elements can be done like dataStruct[1].Id?
EDIT 2:
If I just want a single DataStructure, I can use
public static T ToStruct<T>(byte[] bytes) where T : struct
{
GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
T something = Marshal.PtrToStructure<T>(handle.AddrOfPinnedObject());
handle.Free();
return something;
}
where no copying is required.
EDIT:
The answers possible duplicate are currently about 7 years old, and they consists of either copying the data or implementing a hack.
There is actually a cheat, but it is an ugly unsafe totally unsafe cheat:
[StructLayout(LayoutKind.Sequential)]
//[StructLayout(LayoutKind.Sequential, Pack = 4)]
public struct DataStructure
{
public int Id;
public double Value;
}
[StructLayout(LayoutKind.Explicit)]
public struct DataStructureConverter
{
[FieldOffset(0)]
public int[] IntArray;
[FieldOffset(0)]
public DataStructure[] DataStructureArray;
}
and then you can convert it without problems:
var myarray = new int[8];
myarray[0] = 1;
myarray[3] = 2;
//myarray[4] = 2;
DataStructure[] ds = new DataStructureConverter { IntArray = myarray }.DataStructureArray;
int i1 = ds[0].Id;
int i2 = ds[1].Id;
Note that depending on the size of DataStructure (if it is 16 bytes or 12 bytes), you have to use Pack = 4 (if it is 12 bytes) or you don't need anything (see explanation (1) later)
I'll add that this technique is undocumented and totally unsafe. It even has a problem: ds.Length isn't the length of the DataStructure[] but is the length of the int[] (so in the example given it is 8, not 2)
The "technique" is the same I described here and originally described here.
explanation (1)
The sizeof(double) is 8 bytes, so Value is normally aligned on the 8 bytes boundary, so normally there is a "gap" between Id (that has sizeof(int) == 4) and Value of 4 bytes. So normally sizeof(DataStructure) == 16. Depending on how the DataStructure is built, there could not be this gap, so the Pack = 4 that forces alignment on the 4 byte boundary.
Related
I have to interop with a device, through a C++ interface that accepts byte[] for both in/out streams.
Many of the data being passed around are in the form of an outer struct containing some numbers, and a dynamic array of inner structs (which in turn may contain arrays of their own). A simple form would be:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Outer
{
public float Outer_1;
//...
public double Outer_N;
public Inner[] InnerArray; // variable length
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct Inner
{
public Int16 Inner_1;
//...
public Int32 Inner_N;
}
Simple nested (non-arrayed) structs work fine, either using a fixed(byte*) + Marshal.StructureToPtr(), or GCHandle.Alloc() + Marshal.Copy().
I do know that the array is just a pointer, and its contents would need separate marshalling; but I was hoping I could at least start by marshalling the non-array Outer_1 .. Outer_N part of the structure, followed by a loop to marshal the array contents to different locations of the byte array. I'm trying to avoid having to marshal Outer_1..N one-by-one.
And this is where things fall apart. Neither of the usual methods work.
[Fact]
public unsafe void TestSerialize()
{
Outer managed = new()
{
Outer_1 = 1.0f,
Outer_N = 3.14,
InnerArray = new[] {
new Inner { Inner_1 = 1, Inner_N = Int32.MinValue },
new Inner { Inner_1 = 10, Inner_N = Int32.MaxValue },
},
};
var buffer = new byte[500]; // Could be calculated, but won't bother right now.
//--- Attempt 1:
fixed (byte* pBuf = buffer)
{
//System.NotSupportedException: 'Operation is not supported. (0x80131515)'
Marshal.StructureToPtr(managed, new IntPtr(pBuf), false);
}
//--- Attempt 2:
//System.ArgumentException: 'Object contains non-primitive or non-blittable data'
var managedHandle = GCHandle.Alloc(managed, GCHandleType.Pinned);
Marshal.Copy(
source: managedHandle.AddrOfPinnedObject(),
destination: buffer,
startIndex: 0, length: 12);
managedHandle.Free();
}
Is there a way to marshal the Outer struct without me marshalling Outer_1, Outer_1 ... Outer_N by hand? Probably fine if I'm just dealing with the one, but there's tons of different messages I need to marshal.
I understand that mutable structs are evil. However, I'd still like to compare the performance of an array of structs vs an array of objects. This is what I have so far
public struct HelloStruct
{
public int[] hello1;
public int[] hello2;
public int hello3;
public int hello4;
public byte[] hello5;
public byte[] hello6;
public string hello7;
public string hello8;
public string hello9;
public SomeOtherStruct[] hello10;
}
public struct SomeOtherStruct
{
public int yoyo;
public int yiggityyo;
}
public class HelloClass
{
public int[] hello1;
public int[] hello2;
public int hello3;
public int hello4;
public byte[] hello5;
public byte[] hello6;
public string hello7;
public string hello8;
public string hello9;
public SomeOtherClass[] hello10;
}
public class SomeOtherClass
{
public int yoyo;
public int yiggityyo;
}
static void compareTimesClassVsStruct()
{
HelloStruct[] a = new HelloStruct[50];
for (int i = 0; i < a.Length; i++)
{
a[i] = default(HelloStruct);
}
HelloClass[] b = new HelloClass[50];
for (int i = 0; i < b.Length; i++)
{
b[i] = new HelloClass();
}
Console.WriteLine("Starting now");
var s1 = Stopwatch.StartNew();
for (int i = 0; i < _max; i++)
{
a[i % 50].hello1 = new int[] { 1, 2, 3, 4, i % 50 };
a[i % 50].hello3 = i;
a[i % 50].hello7 = (i % 100).ToString();
}
s1.Stop();
var s2 = Stopwatch.StartNew();
for (int j = 0; j < _max; j++)
{
b[j % 50].hello1 = new int[] { 1, 2, 3, 4, j % 50 };
b[j % 50].hello3 = j;
b[j % 50].hello7 = (j % 100).ToString();
}
s2.Stop();
Console.WriteLine(((double)(s1.Elapsed.TotalSeconds)));
Console.WriteLine(((double)(s2.Elapsed.TotalSeconds)));
Console.Read();
}
There's a couple of things happening here that I'd like to understand.
Firstly, since the array stores structs, when I try to access a struct from the array using the index operation, should I get a copy of the struct or a reference to the original struct? In this case when I inspect the array after running the code, I get the mutated struct values. Why is this so?
Secondly, when I compare the timings inside CompareTimesClassVsStruct() I get approximately the same time. What is the reason behind that? Is there any case under which using an array of structs or an array of objects would outperform the other?
Thanks
When you access the properties of an element of an array of structs, you are NOT operating on a copy of the struct - you are operating on the struct itself. (This is NOT true of a List<SomeStruct> where you will be operating on copies, and the code in your example wouldn't even compile.)
The reason you are seeing similar times is because the times are being distorted by the (j % 100).ToString() and new int[] { 1, 2, 3, 4, j % 50 }; within the loops. The amount of time taken by those two statements is dwarfing the times taken by the array element access.
I changed the test app a little, and I get times for accessing the struct array of 9.3s and the class array of 10s (for 1,000,000,000 loops), so the struct array is noticeably faster, but pretty insignificantly so.
One thing which can make struct arrays faster to iterate over is locality of reference. When iterating over a struct array, adjacent elements are adjacent in memory, which reduces the number of processor cache misses.
The elements of class arrays are not adjacent (although the references to the elements in the array are, of course), which can result in many more processor cache misses while you iterate over the array.
Another thing to be aware of is that the number of contiguous bytes in a struct array is effectively (number of elements) * (sizeof(element)), whereas the number of contiguous bytes in a class array is (number of elements) * (sizeof(reference)) where the size of a reference is 32 bits or 64 bits, depending on memory model.
This can be a problem with large arrays of large structs where the total size of the array would exceed 2^31 bytes.
Another difference you might see in speed is when passing large structs as parameters - obviously it will be much quicker to pass by value a copy of the reference to a reference type on the stack than to pass by value a copy of a large struct.
Finally, note that your sample struct is not very representative. It contains a lot of reference types, all of which will be stored somewhere on the heap, not in the array itself.
As a rule of thumb, structs should not be more than 32 bytes or so in size (the exact limit is a matter of debate), they should contain only primitive (blittable) types, and they should be immutable. And, usually, you shouldn't worry about making things structs anyway, unless you have a provable performance need for them.
Firstly, since the array stores structs, when I try to access a struct from the array using the index operation, should I get a copy of the struct or a reference to the original struct?
Let me tell you what is actually happening rather than answering your confusingly worded either-or question.
Arrays are a collection of variables.
The index operation when applied to an array produces a variable.
Mutating a field of a mutable struct successfully requires that you have in hand the variable that contains the struct you wish to mutate.
So now to your question: Should you get a reference to the struct?
Yes, in the sense that a variable refers to storage.
No, in the sense that the variable does not contain a reference to an object; the struct is not boxed.
No, in the sense that the variable is not a ref variable.
However, if you had called an instance method on the result of the indexer, then a ref variable would have been produced for you; that ref variable is called "this", and it would have been passed to your instance method.
You see how confusing this gets. Better to not think about references at all. Think about variables and values. Indexing an array produces a variable.
Now deduce what would have happened had you used a list rather than an array, knowing that the getter indexer of a list produces a value, not a variable.
In this case when I inspect the array after running the code, I get the mutated struct values. Why is this so?
You mutated a variable.
I get approximately the same time. What is the reason behind that?
The difference is so tiny that it is being swamped by all the memory allocations and memory copies you are doing in both cases. That is the real takeaway here. Are operations on mutable value types stored in arrays slightly faster? Possibly. (They save on collection pressure as well, which is often the more relevant performance metric.) But though the relative savings might be significant, the savings as a percentage of total work is often tiny. If you have a performance problem then you want to attack the most expensive thing, not something that is already cheap.
This question is a follow up from Marshalling C# structure to C++ Using StructureToPtr.
I have the following struct:
[StructLayout(LayoutKind.Explicit, Size = 120, CharSet = CharSet.Unicode)]
public unsafe struct DynamicState
{
[FieldOffset(0)]
public fixed double Position[3];
[FieldOffset(24)]
public fixed double Velocity[3];
[FieldOffset(48)]
public fixed double Acceleration[3];
[FieldOffset(72)]
public fixed double Attitude[3];
[FieldOffset(96)]
public fixed double AngularVelocity[3];
}
If I try and initialise an array like this:
var dynamicState = new DynamicState();
double[] array = new double[] { 1, 2, 3 };
fixed (double* pArray = array)
{
dynamicState.Acceleration = pArray;
}
I get the following error:The left-hand side of an assignment must be a variable, property or indexer.
What is the correct way to initialise an unsafe array that is part of a struct?
Well the simple approach seems to work:
for (int i = 0; i < 3; i++)
{
dynamicState.AngularVelocity[i] = array[i];
}
It may not be as far as you're looking for though. Is this a performance-critical piece of code?
This may be better:
Marshal.Copy(array, 0, new IntPtr(dynamicState.AngularVelocity), array.Length);
I can't say I've much experience with unmanaged code, but it's worth at least looking at those options...
The application is a windows-based C# user interface for an embedded control and monitoring system.
The embedded system (written in C) maintains a table of faults which it sends to the c# application. The fault table contains one bit for each fault, stored in structure. Now from the point of view of the user interface the table is somewhat sparse - There are only a few of the fault bits that we are interested in, so the structure can be represented as below:
typedef struct
{
BYTE FaultsIAmNotInterestedIn0[14];
BYTE PowerSupplyFaults[4];
BYTE FaultsIAmNotInterestedIn1[5];
BYTE MachineryFaults[2];
BYTE FaultsIAmNotInterestedIn2[5];
BYTE CommunicationFaults[4];
}FAULT_TABLE;
Now, I want to be able to index each fault bit that I am interested in. In C I would use an enumeration to do this:
typedef enum
{
FF_PSU1 = offsetof(FAULT_TABLE,PowerSupplyFaults)*8,
FF_PSU2,
FF_PSU3,
FF_PSU4,
FF_PSU5,
FF_PSU6,
FF_PSU7,
FF_PSU8,
FF_PSU9,
FF_PSU10,
FF_PSU11,
FF_PSU12,
FF_PSU13,
FF_MACHINERY1 = offsetof(FAULT_TABLE,MachineryFaults)*8,
FF_MACHINERY2,
FF_MACHINERY3,
FF_MACHINERY4,
FF_MACHINERY5,
FF_MACHINERY6,
FF_MACHINERY7,
FF_MACHINERY8,
FF_MACHINERY9,
FF_MACHINERY10,
FF_COMMS1 = offsetof(FAULT_TABLE,CommunicationFaults)*8,
FF_COMMS2,
FF_COMMS3,
FF_COMMS4,
FF_COMMS5,
FF_COMMS6,
FF_COMMS7,
FF_COMMS8,
FF_COMMS9,
FF_COMMS10,
FF_COMMS11,
FF_COMMS12,
FF_COMMS13
}FAULT_FLAGS;
Is there a way that I can create a similar enumeration, based on a data structure in C#?
Add a Flags attribute to your enum. See http://msdn.microsoft.com/query/dev10.query?appId=Dev10IDEF1&l=EN-US&k=k(SYSTEM.FLAGSATTRIBUTE);k(TargetFrameworkMoniker-%22.NETFRAMEWORK%2cVERSION%3dV4.0%22);k(DevLang-CSHARP)&rd=true
You can get the offset of an element of a struct with Marshal.OffsetOf
http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.marshal.offsetof.aspx
Your struct can be created in pretty much the same way although you need to ensure you specify the layout (although I suppose this depends on how the struct is instantiated).
http://msdn.microsoft.com/en-us/library/system.runtime.interopservices.structlayoutattribute(v=vs.71).aspx
However... the problem will come with the Enum. Enum values need to be compile time constants and Marshal.OffsetOf isn't. So you may have to store the offsets as static members of a class instead.
I also think you'll probably have to stray into unsafe code to make any real use of these offsets once you have them - and there is probably a better way to approach whatever you want to do in managed code.
You can create an enum to map the bits/flags, but you cannot map more than 32 bits.
In your example, I would create several const of long (64 bits). That won't solve the bit limitation when your array is very long (e.g. the first one).
Another trick is creating a static function that gives you the value of the bit (as a bool), upon the bit position in the array. In this case, the position (being a constant) is surely within the int range.
public static bool GetBit(byte[] buffer, int pos)
{
return (buffer[pos >> 3] & (1 << (pos & 7)) != 0);
}
Hope it helps.
Create an enum that specifies the bit-offset within each set of bytes:
enum PowerSupplyFaults
{
PowerSupplyFault1, PowerSupplyFault2, PowerSupplyFault3
}
PowerSupplyFault1 will be auto-assigned 0, Fault2 as 1, etc.
Assuming you receive a struct that mirrors the C version:
struct FaultTable
{
// ...
public byte[] PowerSupplyFaults;
public byte[] MachineFaults;
// ...
}
You can test for a fault by feeding the bytes into a BitArray and testing by index:
FaultTable faults = GetFaultTable();
BitArray psu_faults = new BitArray( fault_table.PowerSupplyFaults );
if ( psu_faults[ (int)PowerSupplyFaults.PowerSupplyFault3 ] )
{
// ...
}
Alternatively, you can walk all the bits and see which are set:
for ( int i = 0; i < psu_faults.Length; i++ )
{
if ( psu_faults[ i ] )
{
Console.WriteLine( "PSU fault: {0}", (PowerSupplyFaults)i );
}
}
I have a struct that looks something like this:
[StructLayout(LayoutKind.Sequential)]
public struct in_addr {
public Anonymous1 S_un;
[StructLayoutAttribute(LayoutKind.Explicit)]
public struct Anonymous1 {
[FieldOffsetAttribute(0)]
public Anonymous2 S_un_b;
[FieldOffsetAttribute(0)]
public Anonymous3 S_un_w;
[FieldOffsetAttribute(0)]
public uint S_addr;
}
[StructLayoutAttribute(LayoutKind.Sequential)]
public struct Anonymous2 {
public byte s_b1;
public byte s_b2;
public byte s_b3;
public byte s_b4;
}
[StructLayoutAttribute(LayoutKind.Sequential)]
public struct Anonymous3 {
public ushort s_w1;
public ushort s_w2;
}
public in_addr(byte[] address) {
S_un.S_addr = (uint)BitConverter.ToInt32(address, 0);
S_un.S_un_b.s_b1 = address[0];
S_un.S_un_b.s_b2 = address[1];
S_un.S_un_b.s_b3 = address[2];
S_un.S_un_b.s_b4 = address[3];
S_un.S_un_w.s_w1 = 0;
S_un.S_un_w.s_w2 = 0;
}
}
When I try to create a new instance of this struct, every byte field is set to 0.
in_addr temp = new in_addr(bytes);
I've stepped through the struct's constructor and verified that the bytes are indeed getting assigned to the fields. But when I step out of the constructor and check the value of temp, everything is 0.
What's causing this?
Because of these lines:
S_un.S_un_w.s_w1 = 0;
S_un.S_un_w.s_w2 = 0;
They are mapped to your 4 bytes via
[FieldOffsetAttribute(0)]
public Anonymous2 S_un_b;
[FieldOffsetAttribute(0)]
public Anonymous3 S_un_w;
I can't reproduce this; your code works for me. I’m using Mono but I very much doubt that this is a bug in the Microsoft C# compiler, it's more likely that you've got an error elsewhere.
I just tested this in VisualStudio 2008 using C# and got the expected output (1, 2).
Try posting the actual example code that you're having an issue with.
EDIT This is why example code can be bad ;)
Your issue was with the StructLayout.Explicit on Anonymous1. Is there a particular reason that
You did explicit instead of sequential and putting them in the order you wanted
You left the indices as 0 for all of them
When I changed it to Sequential and removed the attributes, it worked fine for me.
EDIT 2 (Deleted)
EDIT 3
Your issue is with the assignment in the constructor. Not sure how I missed this. You don't need to set all of those variables, because your FieldOffset attribute makes them all stored in the same location. The struct itself is only occupying 4 bytes of memory (in theory, anyway). Whether you're accessing it via the int, the bytes, or the two shorts, they all go to the same place. As a result, your first two sets of assignments (to the int and to the bytes) are redundant, and the last set (to the shorts setting them to 0) clears out what you just did.
Unfortunately the C# compiler doesn't know this, so I'm sure you added the last assignment because it complained about the struct not being fully assigned. Add the dummy assignments first for the shorts and the int, then assign the bytes explicitly from the array.
If you are using C# 3.0, just try the following:
MyStruct ms = new MyStruct
{
MyByte = 1,
MyNestedStruct.NestedStryctByte = 2
}
That should work, and would alleviate the need to have a constructor in your struct. If you can't directly initialize the NestedStructByte, create it first:
MyStruct ms = new MyStruct
{
MyByte = 1,
MyNestedStruct = new MyStruct.NestedStruct
{
NestedStryctByte = 2
}
}
I tested your code - everything is OK.
public struct MyStruct
{
public byte MyByte;
public NestedStruct MyNestedStruct;
public struct NestedStruct
{
public byte NestedStructByte;
}
public MyStruct(byte[] bytes)
{
MyByte = bytes[0];
MyNestedStruct.NestedStructByte = bytes[1];
}
}
class Program
{
static void Main(string[] args)
{
MyStruct ms = new MyStruct(new byte[] { 1, 2 });
//ms.MyByte; // 0, but should be 1
//ms.MyNestedStruct.NestedStructByte; // 0, but should be 2
}
}