Is there any convention over the algorithm used to make the layouts of structs on C?
I want to be able to have a code running in a vm to be able to have structures compatible with their C counterparts, just like C# interop works. For this I will need to know how the alignment algorithm works. I gather there must be a convetion for that, as it works nicely on C#. I have in mind the probable algorithm they have used to work this out, but I haven't found any proof it is the right one.
Here's how I think it works:
for each declared field (by order of declaration)
See if the field fits in the remaining bytes (until next alignment)
If it doesn't fit, align this field; otherwise add it to current offset
for example, on a 32-bit system for a struct like:
{
byte b1;
byte b2;
int32 i1;
byte b3;
}
would be like this with this algorithm:
{
byte b1;
byte b2;
byte[2] align1;
int32 i1;
byte b3;
byte[3] align2;
}
In general, structure alignment in C depends on the compiler used, and especially the compiler options in effect at the time the structure declaration is processed. You can't make any general assumptions except to say that for a particular structure in a particular program compiled with particular settings, the structure layout can be determined.
That said, your guess closely matches what most compilers are likely to do with default alignment settings.
Related
I'm trying to have my C# struct match up some complex padding and packing rules.
Fields should be aligned on 4 byte boundaries.
The entire struct should be a multiple of 16 bytes
Using the StructLayout attribute I can make sure that fields are aligned on 4 byte boundaries.
[StructLayout(LayoutKind.Sequential, Pack=4)]
struct Foo
{
float a;
float b;
}
But looking at the other options for the StructLayout attribute I see no options for padding the struct to multiples of 16 bytes. Is there no option for that in C#?
The only option I see is manually set the right size, using the Size property of the StructLayout attribute. But that seems brittle to me. As every time somebody adds a field to this struct they should take care not to forget to update the size.
After more searching, it indeed does look like I have to manually set the Size and FieldOffset to get to the right packing rules for Constant Buffers in DirectX/HLSL/C# interop code.
In my own project this is even a bit more complex. Since I use source generators to create these structs. But in the end I was able to figure it out. For those interested the source code can be found here.
In SharpDx BGRA shift for red is 16:
(color >> 16) & 255
see here.
But in .NET, ARGB shift for red is also 16:
private const int ARGBRedShift = 16;
see here and here and here.
I'm confused, what's right?
this (like .NET):
public int PackeColorToArgb()
{
int value = B;
value |= G << 8;
value |= R << 16;
value |= A << 24;
return (int)value;
}
or this (like SharpDx):
public int PackeColorToArgb()
{
int value = A;
value |= B << 8;
value |= G << 16;
value |= R << 24;
return (int)value;
}
For .Net 0xFFFF0000 is Argb Red, but for SharpDx this is Bgra Red. what's right?
The right way to pack BGRA and ARGB color to int
That depends on where you're going to use the int. But for the two examples you've provided, you'll pack the byte values into the integer value in exactly the same way. They are both "right". It's only the name that's different, and you can have many different names for the same thing.
Importantly, your second code example — supposedly "correct" for SharpDx — is not correct for the color format you're asking about. You can see right in the source code you reference, while you are packing the bytes in the order A, B, G, and R, LSB to MSB, the correct order of component is in fact BGRA (again, LSB to MSB).
Your second code example should look just like the first.
Long version…
As you can see from the two source code references you've noted, the actual formats are identical. That is, the format from each API, stores the byte values for each color component in the same order, within a single 32-bit integer: blue in the lowest 8 bits, then green, then red, then alpha in the highest 8 bits.
The problem is, there's no uniform standard for naming such formats. In the context of .NET, they list the color components in big-endian order (which could be thought of as a little ironic, since so much of the Windows ecosystem is based on little-endian hardware…but, see below). I.e. the most-significant byte is listed first: "ARGB". I call this name "big-endian order" simply because that name is consistent with a scenario in which one stores the 32-bit integer in a sequence of 4 bytes in memory on a computer running in big-endian mode. The order of component initials in the name is the same order they'd appear in that context.
On the other hand, in the context of SharpDx, the name is consistent with the order of bytes you'd see on little-endian hardware. The blue byte would come first in memory, then green, red, and finally alpha.
Fact is, both of these are somewhat arbitrary. While most mainstream PCs are running in little-endian mode now, which would argue in favor of the SharpDx naming scheme (which is inherited from the DirectX environment), these APIs both also can be found on big-endian hardware as well, especially as .NET Core is gaining traction. And in a lot of cases, the programmer using the API doesn't even really care what order the bytes are in. For code that has to deal with the individual bytes, it's still important, but a lot of the time it's more about just knowing what format the tools you're using is writing bitmaps in, and then making sure you've specified the correct format to the API.
All that said, I suspect that the main reason for the discrepancy has less to do with big-endian vs little-endian and more to do with underlying philosophical differences between the people responsible for the API. The fact is, even on big-endian hardware, the SharpDx format for a pixel where the components show up in memory in BGRA order will be "BGRA". Because, bitmap formats don't change just because the byte-order mode of the hardware is different. A pixel is not an integer. It's just a sequence of bytes. What will be different is that the shift values will have to be different, i.e. reversed, so that for code that does treat the pixel as a single 32-bit integer, it can access the individual components correctly.
Instead, it seems to me that the .NET designers recognized that the users of their API will most of the time be dealing with colors at a high level (e.g. setting the color of a pen or brush), not at the pixel level of bitmaps, and so naming the pixel format according to conventional order makes more sense. On the other hand, in SharpDx people are much more often dealing with the low-level pixel data, and having a name that reflects the actual byte-wise sequence of components for a pixel makes more sense in that context.
Indeed, the .NET code you've referenced doesn't involve bitmap data. The Color struct is only ever dealing with single int values at a time, with respect to the "ARGB" nomenclature. Since conceptually, we imagine numbers in big-endian format (even for decimal, i.e. with the most-significant digits first), ARGB is more human-readable. On the other hand, in the areas of .NET that do involve byte order in pixel formats, you'll find that the naming goes back to being representative of the actual byte order, e.g. the list of PixelFormats introduced with WPF.
Clear as mud, right? :)
I was comparing Marshal.SizeOf API with sizeof operator in C#. Their outputs for char and bool data types are little surprising. Here are the results:
For Boolean:
Marshal.SizeOf = 4
sizeof = 1
For char:
Marshal.SizeOf = 1
sizeof = 2
On this link from MSDN I got following text:
For all other types, including structs, the sizeof operator can be
used only in unsafe code blocks. Although you can use the
Marshal.SizeOf method, the value returned by this method is not always
the same as the value returned by sizeof. Marshal.SizeOf returns the
size after the type has been marshaled, whereas sizeof returns the
size as it has been allocated by the common language runtime,
including any padding.
I do not know a lot about technicalities of Marshaling but it has something to do with Run-time heuristics when things change. Going by that logic for bool the size changes from 1 to 4. But for char (from 2 to 1) it is just the reverse which is a boomerang for me. I thought for char also it should also increase the way it happened for bool. Can some one help me understand these conflicting behaviors?
Sorry, you really do have to consider the technicalities to make sense of these choices. The target language for pinvoke is the C language, a very old language by modern standards with a lot of history and used in a lot of different machine architectures. It makes very few assumptions about the size of a type, the notion of a byte does not exist. Which made the language very easy to port to the kind of machines that were common back when C was invented and the unusual architectures used in super-computers and digital signal processors.
C did not originally have a bool type. Logical expressions instead use int where a value of 0 represents false and any other value represents true. Also carried forward into the winapi, it does use a BOOL type which is an alias for int. So 4 was the logical choice. But not a universal choice and you have to watch out, many C++ implementations use a single byte, COM Automation chose two bytes.
C does have a char type, the only guarantee is that it has at least 8 bits. Whether it is signed or unsigned is unspecified, most implementations today use signed. Support for an 8-bit byte is universal today on the kind of architectures that can execute managed code so char is always 8 bits in practice. So 1 was the logical choice.
That doesn't make you happy, nobody is happy about it, you can't support text written in an arbitrary language with an 8-bit character type. Unicode came about to solve the disaster with the many possible 8-bit encodings that were in use but it did not have much of an affect on the C and C++ languages. Their committees did add wchar_t (wide character) to the standard but in keeping with old practices they did not nail down its size. Which made it useless, forcing C++ to later add char16_t and char32_t. It is however always 16 bits in compilers that target Windows since that is the operating system's choice for characters (aka WCHAR). It is not in the various Unix flavors, they favor utf8.
That works well in C# too, you are not stuck with 1 byte characters. Every single type in the .NET framework has an implicit [StructLayout] attribute with a CharSet property. The default is CharSet.Ansi, matching the C language default. You can however easily apply your own and pick CharSet.Unicode. You now get two bytes per character, using the utf16 encoding, the string is copied as-is since .NET also uses utf16. Making sure that the native code expects strings in that encoding is however up to you.
I'm doing interop with some native library, which has some non-natural alignment feature which I want to simulate in .NET struct for the layout. Check these two structs:
public struct Int3
{
public int X;
public int Y;
public int Z;
}
public struct MyStruct
{
public short A;
public Int3 Xyz;
public short B;
}
So, within .NET, it uses its own layout rule to create the layout, which is, alignment would be min(sizeof(primitiveSize), StructLayout.Pack). So the layout of MyStruct would be:
[oo--] MyStruct.A (2 bytes data and 2 bytes padding)
[oooo oooo oooo] MyStruct.Xyz (3 int, no padding)
[oo--] MyStruct.B (2 bytes data and 2 bytes padding)
What I want to do is, I want to change the alignment of Int3 to 8 bytes, like something:
[StructLayout(Alignment = 8)]
public struct Int3 { .... }
Then the layout of MyStruct would became:
[oo-- ----] MyStruct.A (2 bytes for data, and 6 bytes padding, to align next Xyz to 8
[oooo oooo oooo ----] MyStruct.Xyz (4 bytes padding for alignment of 8)
[oo-- ----] (6 bytes padding, because the largest alignment in this struct is 8)
So, my question is:
1) Is there such an attribute in .NET to control the non-natural alignment like this?
2) If there is no such built-in attribute, I know there are other attributes such as StructLayout.Explict, OffsetAttribute, StructLayout.Size, StructLayout.Pack. With these attributes, I can simulate this layout manually, but it is not easy to use. So My second question would be, is there a way to hook into .NET struct layout creation which I can interfere the struct layout? What I mean is, I can create a custom attribute to specify the alignment, and then I calculate the layout, but I don't know how to interfere the .NET to use that layout.
Regards, Xiang.
There is no other way to 'hook into .NET' like you want that I am aware of than StructLayout.Explicit (which is just such a mechanism). Interop is quite a specialized need and, beyond the standard WinAPI cases, you should not expect it to be easy. In your case, unless you are dealing with truly large numbers of different structs with this unusual alignment, it's better to spell it out longhand with StructLayout.Explicit on MyStruct.
Almost any structure will be stored as part of a heap object (either as a class field, or as a field of a struct that is stored as a class field, etc.) The .net 32 platform aligns objects on the Large Object Heap to 16-byte boundaries, but other objects to 4-byte boundaries. Unless an object is manually allocated on the LOH, or is an array of more than 999 doubles [due to a truly atrocious hack, IMHO], there is no meaningful way to assure anything more specific than 4-byte alignment. Even if at some moment in time an unpinned struct is 16-byte aligned, any arbitrary GC cycle might relocate it and change that alignment.
I usually program in C++, but for school i have to do a project in C#.
So i went ahead and coded like i was used to in C++, but was surprised when the compiler complained about code like the following:
const uint size = 10;
ArrayList myarray = new ArrayList(size); //Arg 1: cannot convert from 'uint' to 'int
Ok they expect int as argument type, but why ? I would feel much more comfortable with uint as argument type, because uint fits much better in this case.
Why do they use int as argument type pretty much everywhere in the .NET library even if though for many cases negative numbers dont make any sense (since no container nor gui element can have a negative size).
If the reason that they used int is, that they didnt expect that the average user cares about signedness, why didnt they add overloads for uint additonally ?
Is this just MS not caring about sign correctness or are there cases where negative values make some sense/ carry some information (error code ????) for container/gui widget/... sizes ?
I would imagine that Microsoft chose Int32 because UInt32 is not CLS-compliant (in other words not all languages that use the .NET framework support unsigned integers).
Because unsigned integers are not CLS compliant. There are languages that are missing support for them, Java would be an example.
In addition to the answers talking about CLS compliance, consider that integer math (e.g. 10 + 2) results in integer (as in signed) data, which only makes sense; now consider the bother of having to cast every mathematical expression to uint to pass it to one of the methods you refer to.
As for overloads that take uint -- in many cases method arguments are stored as (or used to calculate) property values, which are usually of type int (again for CLS compliance, and possibly for integer math convenience); the discrepancy in sign would be confusing, not to mention vulnerable to overflow.
Stroustrup prefers int over "uint" in The C++ Programming Language, and I think his reasons apply to C# too:
It's about underflow with no warning:
// Unintended very large uint
uint oops = (uint)5 - (uint)10;
or:
// Unintended infinite loop
for( uint counter = 10; counter >= 0; counter-- )
; // Do something
The extra bit of info is rarely worth having to watch for these kinds of bugs.
This possibly come a "little" late, but just found the question and want to add a missing bit.
A prime example where negative values make perfect sense is in graphical frameworks. For sizes, as stated in the question, negatives are out of question, but for position values it's perfectly acceptable to have negative values. Such values make objects to appear off-screen or at least partially cropped:
It follows the very same principle as in mathematics, negative coordinates just make points to go to the opposing from where the axis grows values. Assuming that (0,0) is at the upper-left corner of the screen, negative values displace things to the left and top of that point, making them half-visible.
This is useful for example if you want to implement a scrolling region, where the contents are larger than the available space. Simply all objects positions become negative to begin disappear from the top or larger that the height for disappear from bottom.
Such things aren't limited to C#. Winforms and WPF use that, as per in the question, but most other graphical environments have the same behavior. HTML+CSS can place elements in the same way, or the C/C++ library SDL also can make use of this effect.