I'm trying to fill a structure (does not have to be an actual struct), with data loaded from a byte[].
There are many different data structures in the byte[], one of them is a string, which is declared as:
UInt16 stringLenght
byte[stringLenght] zeroTerminatedString
I 'c' language this could be handled by declaring a fixed size struct, and instead of a the struct containing the actual string, make a pointer to the string.
Something like:
UInt16 stringLength
char* zeroTerminatedString
Is there a (smart) way to do something similar in c#? I mean loading binary data from a file/memory and filling it into a structure?
Regards
Jakob Justesen
That's not how you'd declare it in C. If the record in the file contains a string then you'd declare the structure similar to:
struct Example {
int mumble; // Anything, not necessarily a string length
char text[42];
// etc...
};
The equivalent C# declaration would look like:
[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
private struct Example {
public int mumble;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 42)]
public string text;
// etc...
}
You'd normally use BinaryReader to read the data. But it cannot handle strings like this directly, you have to read them as a byte[] and do the string conversion yourself. You also cannot take advantage of the declarative syntax, you have to write a call for each individual member of the struct.
There's a workaround for that, the Marshal class already knows how to convert unmanaged structures to managed ones with the PtrToStructure() method. Here's a generic implementation, it works for any blittable type. Two versions, a static one that reads from a byte[] and an instance method that was optimized to repeatedly read from a stream. You'd use a FileStream or MemoryStream with that one.
using System;
using System.IO;
using System.Runtime.InteropServices;
class StructTranslator {
public static bool Read<T>(byte[] buffer, int index, ref T retval) {
if (index == buffer.Length) return false;
int size = Marshal.SizeOf(typeof(T));
if (index + size > buffer.Length) throw new IndexOutOfRangeException();
var handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
try {
IntPtr addr = (IntPtr)((long)handle.AddrOfPinnedObject() + index);
retval = (T)Marshal.PtrToStructure(addr, typeof(T));
}
finally {
handle.Free();
}
return true;
}
public bool Read<T>(Stream stream, ref T retval) {
int size = Marshal.SizeOf(typeof(T));
if (buffer == null || size > buffer.Length) buffer = new byte[size];
int len = stream.Read(buffer, 0, size);
if (len == 0) return false;
if (len != size) throw new EndOfStreamException();
var handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
try {
retval = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
}
finally {
handle.Free();
}
return true;
}
private byte[] buffer;
}
Untested, hope it works.
The Marshal should be able to do this for you.
Note that this can only be done to a struct and you will probably need to use the StructLayout attribute.
I'm not 100% sure on how to handle the string or arrays, but BStrWrapper or ArrayWithOffset MIGHT help, also keep on the lookout for similar classes/attributes (I know I have done stuff like this before for binding to native functions).
I read following some where. Can somebody shed some light on it probably with an example
'the message could even be a struct that is just converted to a byte array which is outputted on the debug UART.
Then on the PC side, the incoming byte array can be easily converted back to a struct like object.'
You can use unsafe to access any blittable (arrays, strings, structs etc.) types as a byte pointer.
Local variables of value types don't have to be pinned:
public unsafe void ReadAsBytePointer(MyStruct obj)
{
byte* ptr = (byte*)&obj;
...
}
Arrays must be pinned. The easiest solution is to use fixed:
public unsafe void ReadAsBytePointer(MyStruct[] input)
{
fixed(MyStruct* ptr = input)
{
byte* byteArray = (byte*)ptr;
}
}
For a general case without adding unsafe to your code you can use GCHandle:
static byte[] GetBytes<T>(T input)
where T : struct
{
int size = Marshal.SizeOf(typeof(T));
byte[] result = new byte[size];
GCHandle gc = GCHandle.Alloc(input, GCHandleType.Pinned);
try
{
Marshal.Copy(gc.AddrOfPinnedObject(), result, 0, size);
}
finally
{
gc.Free();
}
return result;
}
I am using the excellent VLC wrapper created by Roman Ginzburg re: nVLC
This part of his code returns a bitmap object.
In myh calling code I then convert it to a byte array.
I sthere a way to directly convert the memory pointer to a byte array with converting to a bitmap object.
this is his code:
unsafe void OnpDisplay(void* opaque, void* picture)
{
lock (m_lock)
{
PixelData* px = (PixelData*)opaque;
MemoryHeap.CopyMemory(m_pBuffer, px->pPixelData, px->size);
m_frameRate++;
if (m_callback != null)
{
using (Bitmap frame = GetBitmap())
{
m_callback(frame);
}
}
}
}
private Bitmap GetBitmap()
{
return new Bitmap(m_format.Width, m_format.Height, m_format.Pitch, m_format.PixelFormat, new IntPtr(m_pBuffer));
}
What I would like is another function like:
private byte[] GetBytes()
{
//not sure what to put here...
}
I am lookinga s I type but still cannot find anything or even if it possible to do so...
Thanks
Use Marshal.Copy. Like this:
private byte[] GetBytes() {
byte[] bytes = new byte[size];
Marshal.Copy(m_pBuffer, bytes, 0, size);
return bytes;
}
I'm not quite sure where you are storing the size of the buffer, but you must know that.
An aside. Why do you write new IntPtr(m_pBuffer) in GetBitmap rather than plain m_pBuffer?
I also wonder why you feel the need to use unsafe code here. Is it really needed?
I am looking for the most efficient/direct way to do this simple C/C++ operation:
void ReadData(FILE *f, uint16 *buf, int startsamp, int nsamps)
{
fseek(f, startsamp*sizeof(uint16), SEEK_SET);
fread(buf, sizeof(uint16), nsamps, f);
}
in C#/.NET. (I'm ignoring return values for clarity - production code would check them.) Specifically, I need to read in many (potentially 10's to 100's of millions) 2-byte (16-bit) "ushort" integer data samples (fixed format, no parsing required) stored in binary in a disk file. The nice thing about the C way is that it reads the samples directly into the "uint16 *" buffer with no CPU involvement, and no copying. Yes, it is potentially "unsafe", as it uses void * pointers to buffers of unknown size, but it seems like there should be a "safe" .NET alternative.
What is the best way to accomplish this in C#? I have looked around, and come across a few hints ("unions" using FieldOffset, "unsafe" code using pointers, Marshalling), but none seem to quite work for this situation, w/out using some sort of copying/conversion. I'd like to avoid BinaryReader.ReadUInt16(), since that is very slow and CPU intensive. On my machine there is about a 25x difference in speed between a for() loop with ReadUInt16(), and reading the bytes directly into a byte[] array with a single Read(). That ratio could be even higher with non-blocking I/O (overlapping "useful" processing while waiting for the disk I/O).
Ideally, I would want to simply "disguise" a ushort[] array as a byte[] array so I could fill it directly with Read(), or somehow have Read() fill the ushort[] array directly:
// DOES NOT WORK!!
public void GetData(FileStream f, ushort [] buf, int startsamp, int nsamps)
{
f.Position = startsamp*sizeof(ushort);
f.Read(buf, 0, nsamps);
}
But there is no Read() method that takes a ushort[] array, only a byte[] array.
Can this be done directly in C#, or do I need to use unmanaged code, or a third-party library, or must I resort to CPU-intensive sample-by-sample conversion? Although "safe" is preferred, I am fine with using "unsafe" code, or some trick with Marshal, I just have not figured it out yet.
Thanks for any guidance!
[UPDATE]
I wanted to add some code as suggested by dtb, as there seem to be precious few examples of ReadArray around. This is a very simple one, w/no error checking shown.
public void ReadMap(string fname, short [] data, int startsamp, int nsamps)
{
var mmf = MemoryMappedFile.CreateFromFile(fname);
var mmacc = mmf.CreateViewAccessor();
mmacc.ReadArray(startsamp*sizeof(short), data, 0, nsamps);
}
Data is safely dumped into your passed array. You can also specify a type for more complex types. It seems able to infer simple types on its own, but with the type specifier, it would look like this:
mmacc.ReadArray<short>(startsamp*sizeof(short), data, 0, nsamps);
[UPATE2]
I wanted to add the code as suggested by Ben's winning answer, in "bare bones" form, similar to above, for comparison. This code was compiled and tested, and works, and is FAST. I used the SafeFileHandle type directly in the DllImport (instead of the more usual IntPtr) to simplify things.
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool ReadFile(SafeFileHandle handle, IntPtr buffer, uint numBytesToRead, out uint numBytesRead, IntPtr overlapped);
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool SetFilePointerEx(SafeFileHandle hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);
unsafe void ReadPINV(FileStream f, short[] buffer, int startsamp, int nsamps)
{
long unused; uint BytesRead;
SafeFileHandle nativeHandle = f.SafeFileHandle; // clears Position property
SetFilePointerEx(nativeHandle, startsamp*sizeof(short), out unused, 0);
fixed(short* pFirst = &buffer[0])
ReadFile(nativeHandle, (IntPtr)pFirst, (uint)nsamps*sizeof(short), out BytesRead, IntPtr.Zero);
}
You can use a MemoryMappedFile. After you have memory-mapped the file, you can create a view (i.e. a MemoryMappedViewAccessor) which provides a ReadArray<T> method. This method can read structs from the file without marshalling, and it works with primitive types lie ushort.
dtb's answer is an even better way (actually, it has to copy the data as well, no gain there), but I just wanted to point out that to extract ushort values from a byte array you should be using BitConverter not BinaryReader
EDIT: example code for p/invoking ReadFile:
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool ReadFile(IntPtr handle, IntPtr buffer, uint numBytesToRead, out uint numBytesRead, IntPtr overlapped);
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool SetFilePointerEx(IntPtr hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);
unsafe bool read(FileStream fs, ushort[] buffer, int offset, int count)
{
if (null == fs) throw new ArgumentNullException();
if (null == buffer) throw new ArgumentNullException();
if (offset < 0 || count < 0 || offset + count > buffer.Length) throw new ArgumentException();
uint bytesToRead = 2 * count;
if (bytesToRead < count) throw new ArgumentException(); // detect integer overflow
long offset = fs.Position;
SafeFileHandle nativeHandle = fs.SafeFileHandle; // clears Position property
try {
long unused;
if (!SetFilePositionEx(nativeHandle, offset, out unused, 0);
fixed (ushort* pFirst = &buffer[offset])
if (!ReadFile(nativeHandle, new IntPtr(pFirst), bytesToRead, out bytesToRead, IntPtr.Zero)
return false;
if (bytesToRead < 2 * count)
return false;
offset += bytesToRead;
return true;
}
finally {
fs.Position = offset; // restore Position property
}
}
I might be a bit late to the game here... but the fastest method I found was using a combination of the previous answers.
If i do the following:
MemoryMappedFile mmf = MemoryMappedFile.CreateFromFile(somePath);
Stream io = mmf.CreateViewStream();
int count;
byte[] byteBuffer = new byte[1024 << 2];
ushort[] dataBuffer = new ushort[buffer.Length >> 1];
while((count = io.Read(byteBuffer, 0, byteBuffer.Length)) > 0)
Buffer.BlockCopy(buffer, 0, dataBuffer, 0, count);
This was ~2x faster than the accepted answer.
For me, the unsafe method was the same as the Buffer.BlockCopy without the MemoryMappedFile. The MemoryMappedFile cut down on a bit of time.
Can anyone tell me how to get a array of bytes into a structure in a direct fashion in C# .NET version 2? Like the familiar fread as found in C, so far I have not had much success in reading a stream of bytes and automatically filling a structure. I have seen some implementations where there is pointer hocus-pocus in the managed code by using the unsafe keyword.
Have a look at this sample:
public unsafe struct foobarStruct{
/* fields here... */
public foobarStruct(int nFakeArgs){
/* Initialize the fields... */
}
public foobarStruct(byte[] data) : this(0) {
unsafe {
GCHandle hByteData = GCHandle.Alloc(data, GCHandleType.Pinned);
IntPtr pByteData = hByteData.AddrOfPinnedObject();
this = (foobarStruct)Marshal.PtrToStructure(pByteData, this.GetType());
hByteData.Free();
}
}
}
The reason I have two constructors in foobarStruct
Is there cannot be an empty constructor.
Pass in a block of memory (as a byte array) into the constructor when instantiating the structure.
Is that implementation good enough or is there a much cleaner way to achieve this?
Edit: I do not want to use the ISerializable interface or its implementation.
I am trying to read a binary image to work out the fields used and determine its data using the PE structures.
There isn't anything wrong with using the P/Invoke marshaller, it is not unsafe and you don't have to use the unsafe keyword. Getting it wrong will just produce bad data. It can be a lot easier to use than explicitly writing the deserialization code, especially when the file contains strings. You can't use BinaryReader.ReadString(), it assumes that the string was written by BinaryWriter. Make sure however that you declare the structure of the data with a struct declaration, this.GetType() is not likely to work out well.
Here's a generic class that will make it work for any structure declaration:
class StructureReader<T> where T : struct {
private byte[] mBuffer;
public StructureReader() {
mBuffer = new byte[Marshal.SizeOf(typeof(T))];
}
public T Read(System.IO.FileStream fs) {
int bytes = fs.Read(mBuffer, 0, mBuffer.Length);
if (bytes == 0) throw new InvalidOperationException("End-of-file reached");
if (bytes != mBuffer.Length) throw new ArgumentException("File contains bad data");
T retval;
GCHandle hdl = GCHandle.Alloc(mBuffer, GCHandleType.Pinned);
try {
retval = (T)Marshal.PtrToStructure(hdl.AddrOfPinnedObject(), typeof(T));
}
finally {
hdl.Free();
}
return retval;
}
A sample declaration for the structure of the data in the file:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
struct Sample {
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 42)]
public string someString;
}
You'll need to tweak the structure declaration and the attributes to get a match with the data in the file. Sample code that reads a file:
var data = new List<Sample>();
var reader = new StructureReader<Sample>();
using (var stream = new FileStream(#"c:\temp\test.bin", FileMode.Open, FileAccess.Read)) {
while(stream.Position < stream.Length) {
data.Add(reader.Read(stream));
}
}
You probably want to use a BinaryReader which allows you to read in primitive types in binary form.
Create a MemoryStream from the byte[] and then use the BinaryReader from that. You should be able to read out the structure and fill in your object accordingly.