Basically I have a binary file which was created using the c++ memory mapped library. I have all the information regarding the struct in c++. I am trying to read that binary file using memory mapped library in c#. Can someone show me any articles/blogs which show how to achieve that?
I am struggling at mapping the c++ struct to c# struct and also the steps required to read and map the content.
Thanks in advance.
First, open the file with a FileStream.
var stream = new FileStream(path, FileMode.Open, FileAccess.Read);
Then instantiate a BinaryReader from that stream. This isn't completely required, but makes working with binary data easier.
var reader = new BinaryReader(stream);
Next, read in as many bytes as the structure needs, as indicated by Marshal.SizeOf:
var bytes = reader.ReadBytes(Marshal.SizeOf(typeof (MyStruct)));
Then, we "pin" that managed memory so the GC doesn't move it, using GCHandle.Alloc:
var handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
Finally, we convert that buffer to a managed structure, using Marshal.PtrToStructure:
var myStruct = (MyStruct)Marshal.PtrToStructure(
handle.AddrOfPinnedObject(), typeof (MyStruct));
Don't forget to free the GCHandle:
handle.Free();
return theStructure;
}
Here is an example of part of that wrapped up in a generic function.
While I haven't tried it, I'm fairly confident this would also work with a using a MemoryMappedFile and calling CreateViewStream (which you pass to the BinaryReader.)
var mmfile = MemoryMappedFile.CreateFromFile(path);
var stream = mmfile.CreateViewStream();
var reader = new BinaryReader(stream);
// ...
Related
I have a requirement where I need to encrypt file of size 1-2 GB in azure function. In am using PGP core library to encrypt file in memory. The below code is throwing out of memory exception if file size is above 700 MB. Note:- I am using azure function. Scaling up of App service plan didn't help.
I there any alternate of Memory stream that I can use. After encryption , I am uploading file into blob storage.
var privateKeyEncoded = Encoding.UTF8.GetString(Convert.FromBase64String(_options.PGPKeys.PublicKey));
using Stream privateKeyStream = StringToStreamUtility.GenerateStreamFromString(privateKeyEncoded);
privateKeyStream.Position = 0;
var encryptionKeys = new EncryptionKeys(privateKeyStream);
var pgp = new PGP(encryptionKeys);
//encrypt stream
var encryptStream = new MemoryStream();
await pgp.EncryptStreamAsync(streamToEncrypt, encryptStream );
MemoryStream is a Stream wrapper over a byte[]` buffer. Every time that buffer is full, a new one with double the size is allocated and the data is copied. This eventually uses double the final buffer size (4GB for a 2GB file) but worse, it results in such memory fragmentation that eventually the memory allocator can't find a new contiguous memory block to allocate. That's when you get an OOM.
While you could avoid OOM errors by specifying a capacity in the constructor, storing 2GB in memory before even starting to write it is very wasteful. With a real FileStream the encrypted bytes would be written out as soon as they were available.
Azure Functions allow temporary storage. This means you can create a temporary file, open a stream on it and use it for encryption.
var tempPath=Path.GetTempFileName();
try
{
using (var outputStream=File.Open(tempPath))
{
await pgp.EncryptStreamAsync(streamToEncrypt, outputStream);
...
}
}
finally
{
File.Delete(tempPath);
}
MemoryStream uses a byte[] internally, and any byte[] is going to get a bit brittle as it gets around/above 1GiB (although in theory a byte[] can be nearly 2 GiB, in reality this isn't a good idea, and is rarely seen).
Frankly, MemoryStream simply isn't a good choice here; I'd probably suggest using a temporary file instead, and use a FileStream. This doesn't attempt to keep everything in memory at once, and is more reliable at large sizes. Alternatively: avoid ever needing all the data at once completely, by performing the encryption in a pass-thru streaming way.
I'm trying to implement a system for a game editor wherein it will load a predefined texture in the library. The texture uses a proprietary format that the library is already capable of reading, but currently only supports doing so using a pointer address and length as input.
I've currently gotten it working by moving the texture into a temporary file, but this is a fairly unoptimized workaround. I was wondering if there was any way in particular to get the address of the internal file, either with or without the stream. Here is my current (working) code, I can provide additional information on how NodeFactory works if necessary but the main thing is that it requires an address and length to work properly.
var resourceName = ("BrawlLib.HardcodedFiles.metal00.tex0");
string listDefault = "";
using (Stream stream = assembly.GetManifestResourceStream(resourceName))
{
string file;
FileStream fstream = new FileStream((file = Path.GetTempFileName()), FileMode.OpenOrCreate);
stream.CopyTo(fstream);
tNode = NodeFactory.FromFile(null, file) as TEX0Node;
Texture.Attach(tNode, _palette);
}
Sorry for this type of question but I am writing a test soon on this and have no clue on the following possible question:
A web server uses the following c#-code fragment to write a static web-object into the socket-object 'sock'. For which type of web-objects does the code work and which it doesn't? With what .Net-class could the code be improved?
...
f = new FileStream(pathName, FileMode.Open, FileAccess.Read);
StreamReader sReader = new StreamReader(f);
sReader.BaseStream.Seek(0, SeekOrigin.Begin);
String s = sReader.ReadlLine();
while (s != null)
{
sock.Send(System.Text.Encoding.ASCII.GetBytes(s));
s = sReader.ReadLine();
}
sReader.Close();
...
What's a "web-object"? I think your teacher made that term up. I assume this means "file".
Anyway, this will fail if the content is not exactly representable as ASCII.
There is no need to go through text at all. Just copy over the bytes:
f.CopyTo(new NetworkStream(sock));
Any other way to copy the bytes unmodified is also fine.
Be aware that you need to wrap resources such as all those streams and sockets into using in order to not leak.
I'm trying to reimplement an existing Matlab 8-band equalizer GUI I created for a project last week in C#. In Matlab, songs load as a dynamic array into memory, where they can be freely manipulated and playing is as easy as sound(array).
I found the NAudio library which conveniently already has Mp3 extractors, players, and both convolution and FFT defined. I was able to open the Mp3 and read all its data into an array (though I'm not positive I'm going about it correctly.) However, even after looking through a couple of examples, I'm struggling to figure out how to take the array and write it back into a stream in such a way as to play it properly (I don't need to write to file).
Following the examples I found, I read my mp3's like this:
private byte[] CreateInputStream(string fileName)
{
byte[] stream;
if (fileName.EndsWith(".mp3"))
{
WaveStream mp3Reader = new Mp3FileReader(fileName);
songFormat = mp3Reader.WaveFormat; // songFormat is a class field
long sizeOfStream = mp3Reader.Length;
stream = new byte[sizeOfStream];
mp3Reader.Read(stream, 0, (int) sizeOfStream);
}
else
{
throw new InvalidOperationException("Unsupported Exception");
}
return stream;
}
Now I have an array of bytes presumably containing raw audio data, which I intend to eventually covert to floats so as to run through the DSP module. Right now, however, I'm simply trying to see if I can play the array of bytes.
Stream outstream = new MemoryStream(stream);
WaveFileWriter wfr = new WaveFileWriter(outstream, songFormat);
// outputStream is an array of bytes and a class variable
wfr.Write(outputStream, 0, (int)outputStream.Length);
WaveFileReader wr = new WaveFileReader(outstream);
volumeStream = new WaveChannel32(wr);
waveOutDevice.Init(volumeStream);
waveOutDevice.Play();
Right now I'm getting errors thrown in WaveFileReader(outstream) which say that it can't read past the end of the stream. I suspect that's not the only thing I'm not doing correctly. Any insights?
Your code isn't working because you never close the WaveFileWriter so its headers aren't written correctly, and you also would need to rewind the MemoryStream.
However, there is no need for writing a WAV file if you want to play back an array of byes. Just use a RawSourceWaveStream and pass in your MemoryStream.
You may also find the AudioFileReader class more suitable to your needs as it will provide the samples as floating point directly, and allow you to modify the volume.
I have a function that is returning MemoryStream array, i want to convert this memory stream array to a FileStream object.
Is it possible if yes can you please provide a way to do that...
Thanks
A.S
You cannot "convert" the stream, because a MemoryStream and a FileStream are very different things. However, you can write the entire contents of the MemoryStream to a file. There is a CopyTo method that you can use for that:
// memStream is the MemoryStream
using (var output = File.Create(filename)) {
memStream.CopyTo(output);
}
A file stream object represents an open file (from disk) as a stream. A memory stream represents an area of memory (byte array) as a stream. So you can't really convert a memory stream into a file stream directly - at least not trivially.
There are two approaches you could take:
OFFLINE: fully consume the contents of the memory stream and write it all out to a file on disk; then open that file as a file stream
ONLINE: extent the FileStream class creating an adapter that will wrap a MemoryStream object and expose it as a FileStream (essentially acting as a converter)
The reason one is marked [OFFLINE] is because you need to have to full contents of the memory stream before you output it to the file (and once you do, modifications to the file stream will not affect the memory stream; nor changes to the memory stream, such as new data, be available to the file stream)
The second one is marked as [ONLINE] because once you create the adapter and you initialize the FileStream object from the MemoryStream you could process any new data in the MemoryStream using the FileStream adapter object. You would essentially be able to read/write and seek into the memory stream using the file stream as a layer on top of the memory stream. Presumably, that's what you'd want to do..
Of course, it depends on what you need to do, but I'm leaning towards the second [ONLINE] version as the better in the general sense.