I am looking for the most efficient/direct way to do this simple C/C++ operation:
void ReadData(FILE *f, uint16 *buf, int startsamp, int nsamps)
{
fseek(f, startsamp*sizeof(uint16), SEEK_SET);
fread(buf, sizeof(uint16), nsamps, f);
}
in C#/.NET. (I'm ignoring return values for clarity - production code would check them.) Specifically, I need to read in many (potentially 10's to 100's of millions) 2-byte (16-bit) "ushort" integer data samples (fixed format, no parsing required) stored in binary in a disk file. The nice thing about the C way is that it reads the samples directly into the "uint16 *" buffer with no CPU involvement, and no copying. Yes, it is potentially "unsafe", as it uses void * pointers to buffers of unknown size, but it seems like there should be a "safe" .NET alternative.
What is the best way to accomplish this in C#? I have looked around, and come across a few hints ("unions" using FieldOffset, "unsafe" code using pointers, Marshalling), but none seem to quite work for this situation, w/out using some sort of copying/conversion. I'd like to avoid BinaryReader.ReadUInt16(), since that is very slow and CPU intensive. On my machine there is about a 25x difference in speed between a for() loop with ReadUInt16(), and reading the bytes directly into a byte[] array with a single Read(). That ratio could be even higher with non-blocking I/O (overlapping "useful" processing while waiting for the disk I/O).
Ideally, I would want to simply "disguise" a ushort[] array as a byte[] array so I could fill it directly with Read(), or somehow have Read() fill the ushort[] array directly:
// DOES NOT WORK!!
public void GetData(FileStream f, ushort [] buf, int startsamp, int nsamps)
{
f.Position = startsamp*sizeof(ushort);
f.Read(buf, 0, nsamps);
}
But there is no Read() method that takes a ushort[] array, only a byte[] array.
Can this be done directly in C#, or do I need to use unmanaged code, or a third-party library, or must I resort to CPU-intensive sample-by-sample conversion? Although "safe" is preferred, I am fine with using "unsafe" code, or some trick with Marshal, I just have not figured it out yet.
Thanks for any guidance!
[UPDATE]
I wanted to add some code as suggested by dtb, as there seem to be precious few examples of ReadArray around. This is a very simple one, w/no error checking shown.
public void ReadMap(string fname, short [] data, int startsamp, int nsamps)
{
var mmf = MemoryMappedFile.CreateFromFile(fname);
var mmacc = mmf.CreateViewAccessor();
mmacc.ReadArray(startsamp*sizeof(short), data, 0, nsamps);
}
Data is safely dumped into your passed array. You can also specify a type for more complex types. It seems able to infer simple types on its own, but with the type specifier, it would look like this:
mmacc.ReadArray<short>(startsamp*sizeof(short), data, 0, nsamps);
[UPATE2]
I wanted to add the code as suggested by Ben's winning answer, in "bare bones" form, similar to above, for comparison. This code was compiled and tested, and works, and is FAST. I used the SafeFileHandle type directly in the DllImport (instead of the more usual IntPtr) to simplify things.
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool ReadFile(SafeFileHandle handle, IntPtr buffer, uint numBytesToRead, out uint numBytesRead, IntPtr overlapped);
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool SetFilePointerEx(SafeFileHandle hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);
unsafe void ReadPINV(FileStream f, short[] buffer, int startsamp, int nsamps)
{
long unused; uint BytesRead;
SafeFileHandle nativeHandle = f.SafeFileHandle; // clears Position property
SetFilePointerEx(nativeHandle, startsamp*sizeof(short), out unused, 0);
fixed(short* pFirst = &buffer[0])
ReadFile(nativeHandle, (IntPtr)pFirst, (uint)nsamps*sizeof(short), out BytesRead, IntPtr.Zero);
}
You can use a MemoryMappedFile. After you have memory-mapped the file, you can create a view (i.e. a MemoryMappedViewAccessor) which provides a ReadArray<T> method. This method can read structs from the file without marshalling, and it works with primitive types lie ushort.
dtb's answer is an even better way (actually, it has to copy the data as well, no gain there), but I just wanted to point out that to extract ushort values from a byte array you should be using BitConverter not BinaryReader
EDIT: example code for p/invoking ReadFile:
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool ReadFile(IntPtr handle, IntPtr buffer, uint numBytesToRead, out uint numBytesRead, IntPtr overlapped);
[DllImport("kernel32.dll", SetLastError=true)]
[return:MarshalAs(UnmanagedType.Bool)]
static extern bool SetFilePointerEx(IntPtr hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);
unsafe bool read(FileStream fs, ushort[] buffer, int offset, int count)
{
if (null == fs) throw new ArgumentNullException();
if (null == buffer) throw new ArgumentNullException();
if (offset < 0 || count < 0 || offset + count > buffer.Length) throw new ArgumentException();
uint bytesToRead = 2 * count;
if (bytesToRead < count) throw new ArgumentException(); // detect integer overflow
long offset = fs.Position;
SafeFileHandle nativeHandle = fs.SafeFileHandle; // clears Position property
try {
long unused;
if (!SetFilePositionEx(nativeHandle, offset, out unused, 0);
fixed (ushort* pFirst = &buffer[offset])
if (!ReadFile(nativeHandle, new IntPtr(pFirst), bytesToRead, out bytesToRead, IntPtr.Zero)
return false;
if (bytesToRead < 2 * count)
return false;
offset += bytesToRead;
return true;
}
finally {
fs.Position = offset; // restore Position property
}
}
I might be a bit late to the game here... but the fastest method I found was using a combination of the previous answers.
If i do the following:
MemoryMappedFile mmf = MemoryMappedFile.CreateFromFile(somePath);
Stream io = mmf.CreateViewStream();
int count;
byte[] byteBuffer = new byte[1024 << 2];
ushort[] dataBuffer = new ushort[buffer.Length >> 1];
while((count = io.Read(byteBuffer, 0, byteBuffer.Length)) > 0)
Buffer.BlockCopy(buffer, 0, dataBuffer, 0, count);
This was ~2x faster than the accepted answer.
For me, the unsafe method was the same as the Buffer.BlockCopy without the MemoryMappedFile. The MemoryMappedFile cut down on a bit of time.
Related
I need to slice a byte[] and pass the sliced section to a C# DLL for processing. I am avoiding Array.Copy, because I am trying not to copy anything to hinder performance. I have been made aware of the ArraySegment class as well as Span and Memory. The confusion I am having is how to actually pass these to the DLL, as I am passing an UnmanagedType.LPArray like so:
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate int ProcessVideoFrame_t([MarshalAs(UnmanagedType.LPArray)] byte[] bytes, uint len);
Is there any way to get an underlying byte[] from these classes, or somehow pass the segment without making a copy?
My current code is:
byte[] bytes = new byte[packet.Payload.Length - headerSize];
Array.Copy(packet.Payload, headerSize, bytes, 0, bytes.Length);
helper.ProcessVideoFrame(CookieHelperDictionary[packet.Header.Cookie], bytes, (uint)bytes.Length);```
unsafe
{
fixed (byte* p = &packet.Payload[headerSize])
{
helper.ProcessVideoFrame(CookieHelperDictionary[packet.Header.Cookie], (IntPtr)p, (uint)(packet.Payload.Length - headerSize));
}
}
I also changed the function definition to this:
[UnmanagedFunctionPointer(CallingConvention.StdCall)]
public delegate int ProcessVideoFrame_t(IntPtr rdh, IntPtr buffer, uint len);
I'm trying to fill a structure (does not have to be an actual struct), with data loaded from a byte[].
There are many different data structures in the byte[], one of them is a string, which is declared as:
UInt16 stringLenght
byte[stringLenght] zeroTerminatedString
I 'c' language this could be handled by declaring a fixed size struct, and instead of a the struct containing the actual string, make a pointer to the string.
Something like:
UInt16 stringLength
char* zeroTerminatedString
Is there a (smart) way to do something similar in c#? I mean loading binary data from a file/memory and filling it into a structure?
Regards
Jakob Justesen
That's not how you'd declare it in C. If the record in the file contains a string then you'd declare the structure similar to:
struct Example {
int mumble; // Anything, not necessarily a string length
char text[42];
// etc...
};
The equivalent C# declaration would look like:
[StructLayout(LayoutKind.Sequential, Pack = 1, CharSet = CharSet.Ansi)]
private struct Example {
public int mumble;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 42)]
public string text;
// etc...
}
You'd normally use BinaryReader to read the data. But it cannot handle strings like this directly, you have to read them as a byte[] and do the string conversion yourself. You also cannot take advantage of the declarative syntax, you have to write a call for each individual member of the struct.
There's a workaround for that, the Marshal class already knows how to convert unmanaged structures to managed ones with the PtrToStructure() method. Here's a generic implementation, it works for any blittable type. Two versions, a static one that reads from a byte[] and an instance method that was optimized to repeatedly read from a stream. You'd use a FileStream or MemoryStream with that one.
using System;
using System.IO;
using System.Runtime.InteropServices;
class StructTranslator {
public static bool Read<T>(byte[] buffer, int index, ref T retval) {
if (index == buffer.Length) return false;
int size = Marshal.SizeOf(typeof(T));
if (index + size > buffer.Length) throw new IndexOutOfRangeException();
var handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
try {
IntPtr addr = (IntPtr)((long)handle.AddrOfPinnedObject() + index);
retval = (T)Marshal.PtrToStructure(addr, typeof(T));
}
finally {
handle.Free();
}
return true;
}
public bool Read<T>(Stream stream, ref T retval) {
int size = Marshal.SizeOf(typeof(T));
if (buffer == null || size > buffer.Length) buffer = new byte[size];
int len = stream.Read(buffer, 0, size);
if (len == 0) return false;
if (len != size) throw new EndOfStreamException();
var handle = GCHandle.Alloc(buffer, GCHandleType.Pinned);
try {
retval = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
}
finally {
handle.Free();
}
return true;
}
private byte[] buffer;
}
Untested, hope it works.
The Marshal should be able to do this for you.
Note that this can only be done to a struct and you will probably need to use the StructLayout attribute.
I'm not 100% sure on how to handle the string or arrays, but BStrWrapper or ArrayWithOffset MIGHT help, also keep on the lookout for similar classes/attributes (I know I have done stuff like this before for binding to native functions).
In a WCE app I’m looking for a way to copy a file (I only have to file name/path of it) to a specific memory address.
The file is rather large’ish, ~40MB, so with limited resources, I was hoping to avoid reading the whole file into memory (byte array), by using the answer from this post:
Copy data from from IntPtr to IntPtr
[DllImport("kernel32.dll", EntryPoint = "CopyMemory", SetLastError = false)]
public static extern void CopyMemory(IntPtr dest, IntPtr src, uint count);
static void Main()
{
const int size = 200;
IntPtr memorySource = Marshal.AllocHGlobal(size);
IntPtr memoryTarget = Marshal.AllocHGlobal(size);
CopyMemory(memoryTarget,memorySource,size);
}
That leaves me to 2 problems.
First of all: How do I assign a memory address to an IntPtr?, kind of like: int* startAddr = &0x00180000.
And secondly: How do I obtain the memory address of a file?
With these two questions answered, my code would look something like:
[DllImport("kernel32.dll", EntryPoint = "CopyMemory", SetLastError = false)]
public static extern void CopyMemory(IntPtr dest, IntPtr src, uint count);
private unsafe void CopyFile()
{
try
{
fixed (Int32* startAddr = /*0x00180000*/)
{
fixed(Int32* fileAddr = /*Memory Address of file*/)
{
CopyMemory(new IntPtr(startAddr), new IntPtr(fileAddr), (uint)new FileInfo("File name").Length);
}
}
}
catch { }
}
Would that be a valid way of going about it?
Any help would be greatly appreciated. Thanks in advance!!
Update:
CopyMemory is not the way of going about it. So please disregard.
Also, Sorry for not being clearer. Basically I want to move a file to the start of a partition on disk. I thought that IntPtr could also point to a disk address, but in retrospect I can see that of course it can not.
Anyway sorry for the confusion.
Files don't have addresses in memory. So what you are asking for is something of a non-sequitur.
So far as I can tell, you don't need any unsafe code at all. You don't need to call CopyMemory. And you don't need to load the entire file in one go and then copy. What you can do is as follows:
Use a standard file stream to read the file.
Read the file in small chunks into a small buffer. Just to be concrete, you might choose to read the file in 8KB chunks.
Each time you read a chunk of the file, copy it to the unmanaged memory location with a call to Marshal.Copy.
The code might look like this:
static void CopyStreamToMemory(Stream stream, IntPtr addr, int bufferSize)
{
byte[] buffer = new byte[bufferSize];
long bytesLeft = stream.Length - stream.Position;
while (bytesLeft > 0)
{
int bytesToCopy = (int)Math.Min(bufferSize, bytesLeft);
stream.Read(buffer, 0, bytesToCopy);
Marshal.Copy(buffer, 0, addr, bytesToCopy);
addr += bytesToCopy;
bytesLeft -= bytesToCopy;
}
}
I have an application that heavily reads and write to files (a custom format), I was told to improve performance by using direct unmanaged code.
Before attempting in the real application I made a small tests just to see how the performance gains would be, but for my surprise, the unmanaged versions seems to be like 8x slower than using simply filestream.
Here is the managed function:
private int length = 100000;
private TimeSpan tspan;
private void UsingManagedFileHandle()
{
DateTime initialTime = DateTime.Now;
using (FileStream fileStream = new FileStream("data2.txt", FileMode.Create, FileAccess.ReadWrite))
{
string line = "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890123";
byte[] bytes = Encoding.Unicode.GetBytes(line);
for (int i = 0; i < length; i++)
{
fileStream.Write(bytes, 0, bytes.Length);
}
fileStream.Close();
}
this.tspan = DateTime.Now.Subtract(initialTime);
label2.Text = "" + this.tspan.TotalMilliseconds + " Milliseconds";
}
Here is the unmanaged way:
public void UsingAnUnmanagedFileHandle()
{
DateTime initialTime;
IntPtr hFile;
hFile = IntPtr.Zero;
hFile = FileInteropFunctions.CreateFile("data1.txt",
FileInteropFunctions.GENERIC_WRITE | FileInteropFunctions.GENERIC_READ,
FileInteropFunctions.FILE_SHARE_WRITE,
IntPtr.Zero,
FileInteropFunctions.CREATE_ALWAYS,
FileInteropFunctions.FILE_ATTRIBUTE_NORMAL,
0);
uint lpNumberOfBytesWritten = 0;
initialTime = DateTime.Now;
if (hFile.ToInt64() > 0)
{
string line = "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890123";
byte[] bytes = Encoding.Unicode.GetBytes(line);
uint bytesLen = (uint)bytes.Length;
for (int i = 0; i < length; i++)
{
FileInteropFunctions.WriteFile(hFile,
bytes,
bytesLen,
out lpNumberOfBytesWritten,
IntPtr.Zero);
}
FileInteropFunctions.CloseHandle(hFile);
this.tspan = DateTime.Now.Subtract(initialTime);
label1.Text = "" + this.tspan.TotalMilliseconds + " Milliseconds";
}
else
label1.Text = "Error";
}
[DllImport("kernel32.dll", SetLastError = true)]
public static extern bool CloseHandle(IntPtr hObject);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern unsafe IntPtr CreateFile(
String lpFileName, // Filename
uint dwDesiredAccess, // Access mode
uint dwShareMode, // Share mode
IntPtr attr, // Security Descriptor
uint dwCreationDisposition, // How to create
uint dwFlagsAndAttributes, // File attributes
uint hTemplateFile); // Handle to template file
[DllImport("kernel32.dll")]
public static extern unsafe int WriteFile(IntPtr hFile,
// byte[] lpBuffer,
[MarshalAs(UnmanagedType.LPArray)] byte[] lpBuffer, // also tried this.
uint nNumberOfBytesToWrite,
out uint lpNumberOfBytesWritten,
IntPtr lpOverlapped);
The iteration using FileStream takes about 70 ms in my computer.
The one using WriteFile takes about 550ms.
I tested several times and with several amount of iterations and the difference in performance is consistent.
I have no idea why the unmanaged code is being slower then the managed code.
EDIT
Thank you very much for your explanations, guys . I thought there was something "magical" undergoing FileStream and you have explained it so well.
So, I know now there's no easy path to gain performance in this part, and I would like to ask you for opinion for other simple ways to gain speed. The file is random access in the real application, and size could range from 1MB to 1GB.
Your unmanaged calls write the data to disk as soon as possible while FileStream is buffered (ie does most operations in-memory and should call the underlying unmanaged calls much less often)
There are constructors on FileStream that let you control the buffer size if you want to tweak performance further.
Well, FileStream is jut a wrapper around CreateFile/WriteFile. It's written by bunch of smart guys. So I see no logical explanation at all why you assume that your one should be faster :P.
As already stated, FileStream probably does extra-buffering before calling WriteFile() thus minimizing unmanaged method calls. And this is important - only make unmanaged calls when they are necessary. They cost. Buffer sizes are usually multiple of disk sector size. You can experiment with different sizes, though this is OS dependent, and most likely will yield other results on other computers.
But it's also important to know that WriteFile() does internal buffering too. It's not like you call WriteFile() and bam it's written to file. It will be flushed to HDD once it's time.
I think there is unnecessary byte[] marshaling going on. Eg when you call WriteFile(), system makes copy of your buffer. It should be avoidable by unsafe() keyword and little bit of hacking.
There is also FILE_FLAG_SEQUENTIAL_SCAN that can't be accessed through FileStream(afaik) and it should let system know that you're gonna do file writes/reads only sequentially. This might give some performance boost theoretically.
The difference is because the calls to WriteFile are synchronous while the writes to the FileStream are not.
By default CreateFile will create a synchronous file handle, so the calls to WriteFile do not return until the data is written. If you add FILE_FLAG_OVERLAPPED to the CreateFile call the un-managed implementation will take approximately the same time as the managed.
See the documenation for Synchronous and Asynchronous I/O Handles section of the CreateFile defini
I am working on reading the FAT32 entry of the hard disk and so far have been successful in reading the entries by making use of the CreateFile, ReadFile, and SetFilePointer APIs. Here is my code (written in C#) so far.
---The DLL IMPORTS-----
[DllImport("kernel32.dll", SetLastError = true)]
static extern IntPtr CreateFile(string lpFileName, Int32 dwDesiredAccess,
Int32 dwShareMode, Int32 lpSecurityAttributes, Int32 dwCreationDisposition,
Int32 dwFlagsAndAttributes, IntPtr hTemplateFile);
[DllImport("kernel32.dll")]
static extern bool ReadFile(IntPtr hFile, byte[] lpBuffer,
uint nNumberOfBytesToRead, out uint lpNumberOfBytesRead, uint lpOverlapped);
[DllImport("kernel32.dll")]
extern static int SetFilePointer(IntPtr hFile, int lDistanceToMove, int lpDistanceToMoveHigh, uint dwMoveMethod);
[DllImport("kernel32.dll")]
extern static Boolean CloseHandle(IntPtr hObject);
------CODE----Will Work in any .NET Application---------
int ret, nread;
IntPtr handle;
int s = 512;
byte[] readbuffer = new byte[512];
IntPtr ptr = CreateFile(#"\\.\F:", -1073741824, 3, 0, 3, 128, IntPtr.Zero);
if (ptr != System.IntPtr.Zero)
{
int i = 100;
int ret = SetFilePointer(ptr, 0, 0, 0);
ret = SetFilePointer(ptr, 4194304, 0, 1);
while (true)
{
byte[] inp = new byte[512];
uint read = 0;
if (ret != -1)
{
ReadFile(ptr, inp, 512, out read, 0);
for (int k = 0; k < 16; k++)
{
string s = ASCIIEncoding.ASCII.GetString(inp, k*32, 11);
if (inp[k*32] == 0xE5)
{
MessageBox.Show(s);
}
}
//ret = SetFilePointer(ptr, 512, 0, 1);
}
}
}
The code above reads the F:\ drive and for trial purposes I have made it to read the first File Directory Cluster and query through each file entry and display the file name if it has been deleted.
However I want to make it to a full-blown application, for which I will have to frequently use the byte array and map it to the specified data structures according to the FAT32 Specification.
How can I efficiently use the byte array into which I am reading the data? I have tried the same code using filestream and binaryreader and it works, however now suppose I have a C Structure something like
struct bios_data
{
byte id[3];
char name[11];
byte sectorpercluster[2];
...
}
I want to have a similar data structure in C# and when I read data to a byte array I want to map it to the structure. I tried many options but didn't get a complete solution. I tried making a class and do serialization too but that also didn't work. I have around 3 more structures like theese which I will be using as I read the data from the FAT entry. How can I best achieve the desired results?
If you want to read binary data directly into structs, C-style, this article may interest you. He wrote an unmanaged wrapper around the C stdio functions and interoperates with it. I have tried it - it does work quite well. It is nice to read directly into a struct in C#, and it is fast. You can just do:
unsafe
{
fmp3.Read<MyStruct>(&myStructVar);
}
I gave an answer on how to convert between byte arrays and structs in this question.