Develop Reference

Marshalling structs with non-aligned arrays

I get an exception when trying to marshal this structure
[StructLayout(LayoutKind.Explicit, Pack = 1)]
public struct Data
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 4, ArraySubType = UnmanagedType.U1)]
[FieldOffset(0x1)]
public byte[] a2;
}
It says
"Could not load type 'WTF.Data' from assembly 'WTF, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' because it contains an object field at offset 1 that is incorrectly aligned or overlapped by a non-object field."
When I change offset 1 to 0 or 4, everything is ok.
What am I doing wrong?
Thanks

The [StructLayout] affects both the managed and the marshaled layout of the struct. Bit of a quirk in .NET but creating blittable structs is a rather big win on interop and the CLR can't ignore the fact that managed code always runs on an entirely unmanaged operating system. Not having to create a copy of a struct but just being able to pass a pointer to the managed version is a very major perf win.
Your [FieldOffset] value violates a very strong guarantee of the .NET memory model, object reference assignments are always atomic. An expensive word that means that another thread can never observe an invalid object reference that is only partially updated. Atomicity requires proper alignment, to a multiple of 4 in 32-bit mode, of 8 in 64-bit mode. It they are misaligned then the processor may need to perform multiple memory bus cycles to glue the bytes together. That's bad, it causes tearing when another thread is also updating the variable. Getting parts of the pointer value from the old value, part from the new. What's left is a corrupted pointer that crashes the garbage collector. Very bad.
Obscure stuff from the high-level point of view of C#, it is however very important to provide basic execution guarantees. You can't get it misaligned to 1, no workaround as long as you use LayoutKind.Explicit. So don't use it.

Please see Hans Passant's answer first - aligned data is a good thing, and the CLR enforces it for a reason. It does seem to be possible to "cheat" though, if you for some reason really need or want to:
[StructLayout( LayoutKind.Sequential, Pack = 1 )]
public struct Data
{
public byte Dummy;
[MarshalAs( UnmanagedType.ByValArray, SizeConst = 4,
ArraySubType = UnmanagedType.U1 )]
public byte[] a2;
}
It is also possible with unsafe code:
[StructLayout( LayoutKind.Explicit, Pack = 1 )]
public unsafe struct Data
{
[FieldOffset( 1 )]
public fixed byte a2[4];
}
But again, probably not a good idea.
Edit: A third option would be to simply make the array 5 bytes long and have it at offset 0, and then just ignore the first byte. This does seem safer.

Marshaling complex nested structures containing booleans

I need to do complex marshaling of several nested structures, containing variable length arrays to other structures, hence I decided to use ICustomMarshaler (see for a good JaredPar's tutorial here). But then I have a problem with a struct defined in C++ as:
typedef struct AStruct{
int32_t a;
AType* b;
int32_t bLength;
bool aBoolean;
bool bBoolean;
};
On the C# side, in the MarshalManagedToNative implementation of ICustomMarshaler I was using:
Marshal.WriteByte(intPtr, offset, Convert.ToByte(aBoolean));
offset += 1;
Marshal.WriteByte(intPtr, offset, Convert.ToByte(bBoolean));
But it was not working since I discovered that each bool in the C++ struct was taking 2 bytes. Indeed in x86 sizeof(AStruct) = 16, not 14. Ok, bool is not guaranteed to take 1 byte and so I tried with unsigned char and uint8_t but still the size is 16.
Now, I know I could use an int32 instead than a boolean, but since I care about the taken space and there are several structs containing boolean that flow to disk (I use HDF5 file format and I want to map those boolean with H5T_NATIVE_UINT8 defined in the HDF5 library that takes 1 byte), is there another way? I mean can I have something inside a struct that is guaranteed to take 1 byte?
EDIT
the same problem applies also to int16 values: depending on how many values are present because of alignment reasons the size of the struct at the end might be different from what expected. On the C# side I do not "see" the C++ struct, I simply write on the unmanaged memory by following the definition of my structs in C++. It is quite a simple process, but if I have instead to think to the real space taken by the struct (either by guessing or by measuring it) it will become more difficult and prone to errors every time I modify the struct.

This answer is in addition to what Hans Passant has said.
It might be easiest to have your structures use a fixed packing size, so you can readily predict the member layout. Keep in mind though that this could affect performance.
The rest of this answer is specific to Microsoft Visual C++, but most compilers offer their own variant of this.
To get you started, check out this SO answer #pragma pack effect and MSDN http://msdn.microsoft.com/en-us/library/2e70t5y1.aspx
What you often use is a pragma pack(push, ...) followed by a pragma pack(pop, ...) idiom to only affect packing for the structures defined between the two pragma's:
#pragma pack(push, 4)
struct someStructure
{
char a;
int b;
...
};
#pragma pack(pop)
This will make someStructure have a predictable packing of 4 byte-alignment of each of its members.
EDIT: From the MSDN page on packing
The alignment of a member will be on a boundary that is either a multiple of n
or a multiple of the size of the member, whichever is smaller.
So for pack(4) a char will be aligned on a 1-byte boundary, a short on a 2-byte, and the rest on a 4-byte boundary.
Which value is best depends on your situation. You'll need to explicitly pack all structures you intend to access, and probably all structures that are members of structures you want to access.

sizeof(AStruct) = 16, not 14
That's correct. The struct has two extra bytes at the end that are not used. They ensure that, if you put the struct in an array, that the fields in the struct are still properly aligned. In 32-bit mode, the int32_t and AType* members require 4 bytes and should be aligned to a multiple of 4 to allow the processor to access them quickly. That can only be achieved if the structure size is a multiple of 4. Thus 14 is rounded up to 16.
Do keep in mind that this does not mean that the bool fields take 2 bytes. A C++ compiler uses just 1 byte for them. The extra 2 bytes are pure padding.
If you use Marshal.SizeOf(typeof(AStruct)) in your C# program then you'll discover that the struct you declared takes 20 bytes. This is not good and the problem you are trying to fix. The bool members are the problem, an issue that goes way, way, back to early versions of the C language. Which did not have a bool type. The default marshaling that the CLR uses is compatible with BOOL, the typedef in the winapi. Which is a 32-bit type.
So you have to be explicit about it when you declare the struct in your C# code, you have to tell the marshaller that you want the 1-byte type. Which you do by declaring the struct member as byte. Or by overriding the default marshaling:
[StructLayout(LayoutKind.Sequential)]
private struct AStruct{
public int a;
public IntPtr b;
public int bLength;
[MarshalAs(UnmanagedType.U1)]
public bool aBoolean;
[MarshalAs(UnmanagedType.U1)]
public bool bBoolean;
}
And you'll now see that Marshal.SizeOf() now returns 16. Do be aware that you have to force your program in 32-bit mode, make sure that the EXE project's Platform Target setting is x86.

VirtualAlloc in c# to allocate large memory

I'm trying to adapt a vendor's c# example code for interfacing with a PCI-Express device. The code basically allocates a large buffer as an int array, and then pins it via the fixed keyword before handing it off to hardware to be filled with data.
This works great, but it eventually fails because .Net is limited to ~2 billion elements in an array. I can push the limit out to 16 GB by using an array of Long and gcAllowVeryLargeObjects keyword, but eventually I still run into .Net limitations.
In unmanaged code I could call VirtualAlloc and request 40 or 50GB directly, however its not clear to me if this is possible in c# and I haven't been able to find any good example code. I realize I could be doing this in a different language, but on Windows at least I'm more familiar with .Net, and aside from this relatively small portion of the program, there is very little hardware-specific code so I'd like to try and stick with what I have.

You can pinvoke VirtualAlloc. The signature is
[DllImport("kernel32.dll", SetLastError=true)]
static extern IntPtr VirtualAlloc(IntPtr lpAddress, UIntPtr dwSize, AllocationType lAllocationType, MemoryProtection flProtect);
You can find most pinvoke structures and signatures on pinvoke.net: VirtualAlloc
Alternatively, take a look at the AllocHGlobal func

gcAllowVeryLargeObjects should work just fine, are you sure it doesn't? You may have to explicitly set the target CPU to x64 only.
In any case, you could use a hack struct to get a large value-type (which you can use as an array):
unsafe struct Test
{
public fixed byte Data[1024];
}
unsafe void Main()
{
Test[] test = new Test[16 * 1024 * 1024];
// We've got 16 * 1024 * 1024 * 1024 here.
fixed (Test* pTest = test)
{
}
}
This does have its limits (the unsafe structure has a maximum size), but it should get you where you need to be.
However, it might be better idea to simply call VirtualAlloc through P/Invoke. Or better, use Marshal.AllocHGlobal, it should really be doing the same thing (although you can't specify any parameters except for the size).

Difference between IntPtr and UIntPtr

I was looking at the P/Invoke declaration of RegOpenKeyEx when I noticed this comment on the page:
Changed IntPtr to UIntPtr: When invoking with IntPtr for the handles, you will run into an Overflow. UIntPtr is the right choice if you wish this to work correctly on 32 and 64 bit platforms.
This doesn't make much sense to me: both IntPtr and UIntPtr are supposed to represent pointers so their size should match the bitness of the OS - either 32 bits or 64 bits. Since these are not numbers but pointers, their signed numeric values shouldn't matter, only the bits that represent the address they point to. I cannot think of any reason why there would be a difference between these two but this comment made me uncertain.
Is there a specific reason to use UIntPtr instead of IntPtr? According to the documentation:
The IntPtr type is CLS-compliant, while the UIntPtr type is not. Only the IntPtr type is used in the common language runtime. The UIntPtr type is provided mostly to maintain architectural symmetry with the IntPtr type.
This, of course, implies that there's no difference (as long as someone doesn't try to convert the values to integers). So is the above comment from pinvoke.net incorrect?
Edit:
After reading MarkH's answer, I did a bit of checking and found out that .NET applications are not large address aware and can only handle a 2GB virtual address space when compiled in 32-bit mode. (One can use a hack to turn on the large address aware flag but MarkH's answer shows that checks inside the .NET Framework will break things because the address space is assumed to be only 2GB, not 3GB.)
This means that all correct virtual memory addresses a pointer can have (as far as the .NET Framework is concerned) will be between 0x00000000 and 0x7FFFFFFF. When this range is translated to signed int, no values would be negative because the highest bit is not set. This reinforces my belief that there's no difference in using IntPtr vs UIntPtr. Is my reasoning correct?
Fermat2357 pointed out that the above edit is wrong.

UIntPtr and IntPtr are internal implemented as
private unsafe void* m_value;
You are right both simply only managing the bits that represent a address.
The only thing where I can think about an overflow issue is if you try to perform pointer arithmetics. Both classes support adding and subtracting of offsets. But also in this case the binary representation should be ok after such an operation.
From my experience I would also prefer UIntPtr, because I think on a pointer as an unsigned object. But this is not relevant and only my opinion.
It seems not make any difference if you use IntPtr or UIntPtr in your case.
EDIT:
IntPtr is CLS-compliant because there are languages on top of the CLR which not support unsigned.

This, of course, implies that there's no difference (as long as someone doesn't try to convert the values to integers).
Unfortunately, the framework attempts to do precisely this (when compiled specifically for x86). Both the IntPtr(long) constructor, and the ToInt32() methods attempt to cast the value to an int in a checked expression. Here's the implementation seen if using the framework debugging symbols.
public unsafe IntPtr(long value)
{
#if WIN32
m_value = (void *)checked((int)value);
#else
m_value = (void *)value;
#endif
}
Of course, the checked expression will throw the exception if the value is out of bounds. The UIntPtr doesn't overflow for the same value, because it attempts to cast to uint instead.

The difference between IntPtr\UIntPtr is the same as the differences found between: Int32\UInt32 (ie, it's all about how the numbers get interpreted.)
Usually, it doesn't matter which you choose, but, as mentioned, in some cases, it can come back to bite you.
(I'm not sure why MS chose IntPtr to begin with(CLS Compliance, etc,.), memory is handled as a DWORD(u32), meaning it's unsigned, thus, the preferred method should be UIntPtr, not IntPtr, right?)
Even: UIntPtr.Add()
Seems wrong to me, it takes a UIntPtr as a pointer, and an 'int' for the offset. (When, to me, 'uint' would make much more sense. Why feed a signed value to an unsigned method, when, more than likely the code cast it to 'uint' under the hood. /facepalm)
I would personally prefer UIntPtr over IntPtr simply because the unsigned values match the values of the underlying memory which I'm working with. :)
Btw, I'll likely end up creating my own pointer type(using UInt32), built specifically for working directly with memory. (I'm guessing that UIntPtr isn't going to catch all the possible bad memory issues, ie, 0xBADF00D, etc, etc,. Which is a CTD waiting to happen... I'll have to see how the built in type handles things first, hopefully, a null\zero check is properly filtering out stuff like this.)

CLS-compliant alternative for ulong property

Background
I am writing a managed x64 assembler (which is also a library), so it has multiple classes which define an unsigned 64-bit integer property for use as addresses and offsets. Some are file offsets, others are absolute addresses (relative to the main memory) and again others are relative virtual addresses.
Problem
I use the ulong datatype for the properties in the mentioned scenarios, and this works fine. However, such properties are not CLS-compliant. I can mark them as [ClsCompliant(false)], but then I need to provide a CLS-compliant alternative to users of the library.
Options and questions
Some suggest providing an alternative property with a bigger data type, but this is not an option because there is no bigger signed integer primitive which could hold all values from 0 to UInt64.MaxValue.
I would rather not mark my entire assembly as non-CLS-compliant, because in most usage scenario's, not all the possible values up to UInt64.MaxValue are used. So, for e.g. Address I could provide an alternative long property AddressAlternative, which only accepts positive values. However, what should happen when Address somehow contains a value above Int64.MaxValue. Should AddressAlternative throw some exception?
And what would be an appropriate name for AddressAlternative?
Providing an alternative for every usage of ulong would result in many 'double' properties. Is there a better way to do this? Note that not all usages of ulong properties have the same semantics, so a single struct would not cut it.
And finally, I have the same CLS compliance problem in constructor parameters. So should I provide an alternative overload accepting long for such a parameter?
I do not mind restricting the use of (some functionality) of the library when it is used from a CLS-only context, as long as it can be used in most scenarios.

but when it represents an unsigned address above Int64.MaxValue
You are using the wrong type, addresses must be stored in IntPtr or UIntPtr. There is just no way your problem is realistic. If you can't afford to lose the single bit in UInt64 then you are way too close to overflow. If this number represents an index then a plain Int32 will be fine, .NET memory blobs are limited to 2 gigabyte, even on a 64-bit machine.
If it is an address then IntPtr will be fine for a very, very long time. Currently available hardware is 4.5 orders of magnitude away from reaching that limit. Very drastic hardware redesign will be needed to get close, you'll have much bigger problems to worry about when that day ever comes. Nine exabyte of virtual memory is enough for everybody until I retire.

Microsoft defines a 64-bit address as Int64, not UInt64, so you can still be CLS compliant.
Please refer to http://msdn.microsoft.com/en-us/library/837ksy6h.aspx.
Which basically says:
IntPtr Constructor (Int64)
Initializes
a new instance of IntPtr using the
specified 64-bit pointer.
Parameters
value
Type: System.Int64
A pointer or handle contained in a 64-bit signed integer.
And yes, I just did a quick test and the following worked fine in a project targeted for either x64 or Any CPU. I placed a brekpoint in the code and examined x. However, when targeted for only x86, it will throw an exception.
using System;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
IntPtr x = new IntPtr(long.MaxValue);
}
}
}
But, if it turns out that you really need that extra bit. You could provide two libraries. One that is CLS-Compliant and one that is not--user's choice. This could be accomplished by using #if statements and using Conditional Compilation Symbols. This way, you can define the same variable name, but with different definitions. http://msdn.microsoft.com/en-us/library/4y6tbswk.aspx