Is there a way to read binary data from file into an array like in C where I can pass a pointer of any type to the i/o functions? I am thinking of something like BinaryReader::ReadBytes(), but that returns a byte[] which I cannot cast to the desired array pointer type.
If you have a fixed size struct
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
struct MyFixedStruct
{
//..
}
You can then read it in in one go using this:
public static T ReadStruct<T>(Stream stream)
{
byte[] buffer = new byte[Marshal.SizeOf(typeof(T))];
stream.Read(buffer, 0, Marshal.SizeOf(typeof(T)));
GCHandle handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
T typedStruct = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
handle.Free();
return typedStruct;
}
This reads in a byte array covering the size of the struct and then marshals the byte array into the structure. You can use it like this:
MyFixedStruct fixedStruct = ReadStruct<MyFixedStruct>(stream);
The struct may include array types as long as the array length is specified, i.e:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
public struct MyFixedStruct
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 5)]
public int[] someInts; // 5 int's
//..
};
Edit:
I see you just want to read in a short array - In this case just read in the byte array and use Buffer.BlockCopy() to convert to the array you want:
byte[] someBytes = ..
short[] someShorts = new short[someBytes.Length/2];
Buffer.BlockCopy(someBytes, 0, someShorts, 0, someBytes.Length);
This is quite efficient, equivalent to a memcpy in C++ under the hood. The only other overhead you have of course is that the original byte array will be allocated and later garbage collected. This approach would work for any other primitive array type as well.
How about storing a serialized array of your struct in the file? You can build the array of structs easily. Not sure how to stream through the file though, as you would do in C.
How about using a Stream Class as it provides a generic view for sequence of bytes.
Related
I have the following nested structures.
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct ERROR_ITEM
{
byte ErrorID;
};
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct ERROR_DATA
{
[MarshalAs(UnmanagedType.ByValArray, ArraySubType = UnmanagedType.Struct, SizeConst = 10)]
ERROR_ITEM[] ErrorItem;
};
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct VCP_DATA
{
[MarshalAs(UnmanagedType.Struct)]
ERROR_DATA ErrorData;
};
I need to copy a byte array to this structure, so I tried the following
vcpBuffer = new VCP_DATA();
GCHandle handle = GCHandle.Alloc(vcpBuffer, GCHandleType.Pinned);
try
{
IntPtr pBuffer = handle.AddrOfPinnedObject();
Marshal.Copy(bytarray, 0, pBuffer, length);
}
finally
{
if (handle.IsAllocated)
handle.Free();
}
But GCHandle.Alloc() returns the error "An unhandled exception of type System.Argument.Execption" occurred in mscorlib.dll.
Additional information: Object contains non-primitive or non-blittable data.
vcpBuffer = new VCP_DATA();
GCHandle handle = GCHandle.Alloc(bytearray, GCHandleType.Pinned);
try
{
IntPtr pBuffer = handle.AddrOfPinnedObject();
vcpBuffer = (VCP_DATA)Marshal.PtrToStructure(pBuffer, typeof(VCP_DATA));
}
finally
{
if (handle.IsAllocated)
handle.Free();
}
First of all, ERROR_ITEM[] is a managed array, so that's not a blittable structure. It's just a managed reference. The memory that reference points to has a syncblock, method table pointer, and a length specifier sitting in front of the actual elements.
However, using 'fixed' (https://msdn.microsoft.com/en-us/library/zycewsya.aspxhttps://msdn.microsoft.com/en-us/library/zycewsya.aspx) isn't going to help (but check me on that). To get past this error, since ERROR_ITEM[] is of a fixed length, just replace the array with 16 of those ERROR_ITEM fields. You can still use array syntax against the address of the first ERROR_ITEM (ERROR_ITEM*) to access subsequent elements.
Alternately, just compute the size of all 16 elements, but include only the first one as a field, then specify the Size parameter on the StructLayout attribute for ERROR_DATA so that it's big enough to hold them all.
Also, Resharper sometimes whines about nested stuff when the actual compiler is perfectly happy with it. But this is caused by it being an array. Even a fixed unsafe embdedded array makes C# think it's unblittable in my experience.
I've found questions such as this one, which have come close to solving my dilemma. However, I've yet to find a clean approach to solving this issue in a generic manner.
I have a project that has a lot of structs that will be used for binary data transmission. This data needs to be Big Endian and, of course, most .Net architecture is Little Endian. This means that when I convert my structs to bytes, the byte order for my values are reversed.
Is there a fairly straight-forward approach to either forcing my structs to contain data in Big Endian format, or is there a way to generically write these structs to byte arrays (and byte arrays to structs) that output Big Endian data?
Here is some sample code for what I've already done.
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public unsafe struct StructType_1
{
short shortVal;
ulong ulongVal;
int intVal;
}
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public unsafe struct StructType_2
{
long longVal_1;
long longVal_2;
long longVal;
int intVal;
}
...
public static class StructHelper
{
//How can I change the following methods so that the struct data
//is converted to and from BigEndian data?
public static byte[] StructToByteArray<T>(T structVal) where T : struct
{
int size = Marshal.SizeOf(structVal);
byte[] arr = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(structVal, ptr, true);
Marshal.Copy(ptr, arr, 0, size);
Marshal.FreeHGlobal(ptr);
return arr;
}
public static T StructFromByteArray<T>(byte[] bytes) where T : struct
{
int sz = Marshal.SizeOf(typeof(T));
IntPtr buff = Marshal.AllocHGlobal(sz);
Marshal.Copy(bytes, 0, buff, sz);
T ret = (T)Marshal.PtrToStructure(buff, typeof(T));
Marshal.FreeHGlobal(buff);
return ret;
}
}
If you don't mind reading and writing each field to a stream (which may have performance implications) you could use Jon Skeet's EndianBinaryWriter: https://stackoverflow.com/a/1540269/106159
The code would look something like this:
public unsafe struct StructType_2
{
long longVal_1;
long longVal_2;
long longVal;
int intVal;
}
using (MemoryStream memory = new MemoryStream())
{
using (EndianBinaryWriter writer = new EndianBinaryWriter(EndianBitConverter.Big, stream))
{
writer.Write(longVal_1);
writer.Write(longVal_2);
writer.Write(longVal);
writer.Write(intVal);
}
byte[] buffer = memory.ToArray();
// Use buffer
}
You would use the EndianBinaryReader for data going in the opposite direction.
This does of course have two fairly large drawbacks:
It's fiddly writing code to read and write each field.
It may be too slow to do it this way, depending on performance requirements.
Also have a look at this answers to this similar question: Marshalling a big-endian byte collection into a struct in order to pull out values
Given the example on the BitConverter that shows converting a uint, I would suspect not.
Greetings Overflowers,
I love the flexibility of memory mapped files in that you can read/write any value type.
Is there a way to do the same with byte arrays without having to copy them into for e.g. a memory map buffers ?
Regards
You can use the BitConverter class to convert between base data types and byte arrays.
You can read values directly from the array:
int value = BitConverter.ToInt32(data, pos);
To write data you convert it to a byte array, and copy it into the data:
BitConverter.GetBytes(value).CopyTo(data, pos);
You can bind a MemoryStream to a given byte array, set it's property Position to go to a specific position within the array, and then use a BinaryReader or BinaryWriter to read / write values of different types from/to it.
You are searching the MemoryStream class which can be initialised (without copying!) from a fixed-size byte array.
(Using unsafe code)
The following sample shows how to fill a 16 byte array with two long values, which is something BitConverter still can't do without an additional copy operation:
byte[] bar = new byte[16];
long lValue1 = 1;
long lValue2 = 2;
unsafe {
fixed (byte* bptr = &bar[0]) {
long* lptr = (long*)bptr;
*lptr = lValue1;
// pointer arithmetic: for a long* pointer '+1' adds 8 bytes.
*(lptr + 1) = lValue2;
}
}
Or you could make your own StoreBytes() method:
// here the dest offset is in bytes
public static void StoreBytes(long lValue, byte[] dest, int iDestOffset) {
unsafe {
fixed (byte* bptr = &dest[iDestOffset]) {
long* lptr = (long*)bptr;
*lptr = lValue;
}
}
}
Reading values from a byte array is no problem with BitConverter since you can specify the offset in .ToInt64.
Alternative : use Buffer.BlockCopy, which can convert between array types.
I have a sequential struct that I'd like to serialize to a file, which seems trivial. However, this struct consists of, among other things, 2 arrays of other types of structs. The main struct is defined as follows...
[StructLayout(LayoutKind.Sequential)]
public struct ParentStruct
{
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 256)]
public const string prefix = "PRE";
public Int32 someInteger;
public DataLocater[] locater; //DataLocater is another struct
public Body[] body; //Body is another struct
};
I can create these structs exactly as intended. However, when trying to serialize with the following method (which seems popular online), I get an AccessViolationException:
public static byte[] RawSerialize(object structure)
{
int size = Marshal.SizeOf(structure);
IntPtr buffer = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(structure, buffer, true);
byte[] data = new byte[size];
Marshal.Copy(buffer, data, 0, size);
Marshal.FreeHGlobal(buffer);
return data;
}
I'm assuming this is because the structures don't define exactly how large the arrays are, so it cannot explicitly determine the size beforehand? It seems that since it cannot get that, it is not allocating the right amount of space for the structure and it ends up being too short when casting the structure to a pointer. I'm not sure on this. Why might this occur and what are possible alternatives?
Edit: The line throwing the error is
Marshal.StructureToPtr(structure, buffer, true);
Not possible because of the nested struct arrays. See When I try to use a structure containing an array of other structures, I get an exception. What's wrong?.
In C# it makes more sense to implement ISerializable and use the BinaryFormatter class to write the struct to disk.
ISerializable
I'm trying to read binary data using C#. I have all the information about the layout of the data in the files I want to read. I'm able to read the data "chunk by chunk", i.e. getting the first 40 bytes of data converting it to a string, get the next 40 bytes.
Since there are at least three slightly different version of the data, I would like to read the data directly into a struct. It just feels so much more right than by reading it "line by line".
I have tried the following approach but to no avail:
StructType aStruct;
int count = Marshal.SizeOf(typeof(StructType));
byte[] readBuffer = new byte[count];
BinaryReader reader = new BinaryReader(stream);
readBuffer = reader.ReadBytes(count);
GCHandle handle = GCHandle.Alloc(readBuffer, GCHandleType.Pinned);
aStruct = (StructType) Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(StructType));
handle.Free();
The stream is an opened FileStream from which I have began to read from. I get an AccessViolationException when using Marshal.PtrToStructure.
The stream contains more information than I'm trying to read since I'm not interested in data at the end of the file.
The struct is defined like:
[StructLayout(LayoutKind.Explicit)]
struct StructType
{
[FieldOffset(0)]
public string FileDate;
[FieldOffset(8)]
public string FileTime;
[FieldOffset(16)]
public int Id1;
[FieldOffset(20)]
public string Id2;
}
The examples code is changed from original to make this question shorter.
How would I read binary data from a file into a struct?
The problem is the strings in your struct. I found that marshaling types like byte/short/int is not a problem; but when you need to marshal into a complex type such as a string, you need your struct to explicitly mimic an unmanaged type. You can do this with the MarshalAs attrib.
For your example, the following should work:
[StructLayout(LayoutKind.Explicit)]
struct StructType
{
[FieldOffset(0)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 8)]
public string FileDate;
[FieldOffset(8)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 8)]
public string FileTime;
[FieldOffset(16)]
public int Id1;
[FieldOffset(20)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 66)] //Or however long Id2 is.
public string Id2;
}
Here is what I am using.This worked successfully for me for reading Portable Executable Format.It's a generic function, so T is your struct type.
public static T ByteToType<T>(BinaryReader reader)
{
byte[] bytes = reader.ReadBytes(Marshal.SizeOf(typeof(T)));
GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
T theStructure = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
handle.Free();
return theStructure;
}
As Ronnie said, I'd use BinaryReader and read each field individually. I can't find the link to the article with this info, but it's been observed that using BinaryReader to read each individual field can be faster than Marshal.PtrToStruct, if the struct contains less than 30-40 or so fields. I'll post the link to the article when I find it.
The article's link is at: http://www.codeproject.com/Articles/10750/Fast-Binary-File-Reading-with-C
When marshaling an array of structs, PtrToStruct gains the upper-hand more quickly, because you can think of the field count as fields * array length.
I don't see any problem with your code.
just out of my head, what if you try to do it manually? does it work?
BinaryReader reader = new BinaryReader(stream);
StructType o = new StructType();
o.FileDate = Encoding.ASCII.GetString(reader.ReadBytes(8));
o.FileTime = Encoding.ASCII.GetString(reader.ReadBytes(8));
...
...
...
also try
StructType o = new StructType();
byte[] buffer = new byte[Marshal.SizeOf(typeof(StructType))];
GCHandle handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
Marshal.StructureToPtr(o, handle.AddrOfPinnedObject(), false);
handle.Free();
then use buffer[] in your BinaryReader instead of reading data from FileStream to see whether you still get AccessViolation exception.
I had no luck using the
BinaryFormatter, I guess I have to
have a complete struct that matches
the content of the file exactly.
That makes sense, BinaryFormatter has its own data format, completely incompatible with yours.
I had no luck using the BinaryFormatter, I guess I have to have a complete struct that matches the content of the file exactly. I realised that in the end I wasn't interested in very much of the file content anyway so I went with the solution of reading part of stream into a bytebuffer and then converting it using
Encoding.ASCII.GetString()
for strings and
BitConverter.ToInt32()
for the integers.
I will need to be able to parse more of the file later on but for this version I got away with just a couple of lines of code.
Try this:
using (FileStream stream = new FileStream(fileName, FileMode.Open))
{
BinaryFormatter formatter = new BinaryFormatter();
StructType aStruct = (StructType)formatter.Deserialize(filestream);
}
Reading straight into structs is evil - many a C program has fallen over because of different byte orderings, different compiler implementations of fields, packing, word size.......
You are best of serialising and deserialising byte by byte. Use the build in stuff if you want or just get used to BinaryReader.
I had structure:
[StructLayout(LayoutKind.Explicit, Size = 21)]
public struct RecordStruct
{
[FieldOffset(0)]
public double Var1;
[FieldOffset(8)]
public byte var2
[FieldOffset(9)]
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 12)]
public string String1;
}
}
and I received "incorrectly aligned or overlapped by non-object".
Based on that I found:
https://social.msdn.microsoft.com/Forums/vstudio/en-US/2f9ffce5-4c64-4ea7-a994-06b372b28c39/strange-issue-with-layoutkindexplicit?forum=clr
OK. I think I understand what's going on here. It seems like the
problem is related to the fact that the array type (which is an object
type) must be stored at a 4-byte boundary in memory. However, what
you're really trying to do is serialize the 6 bytes separately.
I think the problem is the mix between FieldOffset and serialization
rules. I'm thinking that structlayout.sequential may work for you,
since it doesn't actually modify the in-memory representation of the
structure. I think FieldOffset is actually modifying the in-memory
layout of the type. This causes problems because the .NET framework
requires object references to be aligned on appropriate boundaries (it
seems).
So my struct was defined as explicit with:
[StructLayout(LayoutKind.Explicit, Size = 21)]
and thus my fields had specified
[FieldOffset(<offset_number>)]
but when you change your struct to Sequentional, you can get rid of those offsets and the error will disappear. Something like:
[StructLayout(LayoutKind.Sequential, Size = 21)]
public struct RecordStruct
{
public double Var1;
public byte var2;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 12)]
public string String1;
}
}