I have been using this code to capture the webcam and I have been trying to learn from it and make it better. Rider IDE suggested I should use an async variant of MemoryMappedViewStream.Read but it doesn't work at all. It produces all-black images suggesting the async and sync methods are totally different. I am wondering why that's the case?
// Working:
sourceStream.Read(MemoryMarshal.AsBytes(image.GetPixelMemoryGroup().Single().Span));
// NOT Working:
var bytes = MemoryMarshal.AsBytes(image.GetPixelMemoryGroup().Single().Span).ToArray();
await sourceStream.ReadAsync(bytes, 0, bytes.Length, token);
Repository and line of code
Those two versions are not the same. In "sync" version you obtain a reference to memory location of an image via image.GetPixelMemoryGroup(). Then you read data from sourceStream directly into that location.
In "async" version you again obtain reference to memory location via image.GetPixelMemoryGroup but then you do something different - you call ToArray. This extension method copies bytes from image memory location into new array, the one you hold in bytes variable. You then read data from sourceStream into that bytes array, NOT directly into image memory locaiton. Then you discard bytes array, so you read them to nowhere basically.
Now,MemoryMappedViewStream inherits from UnmanagedMemoryStream and all read\write operations are implemented in UnmanagedMemoryStream. This kind of stream represents data in memory and there is nothing async it can do. The only reason it even has ReadAsync is because base stream class (Stream) has those methods. Even if you manage to make ReadAsync work - in this case it will not be asynchornous anyway. As far as I know - MemoryMappedViewStream does now allow real asynchronous access, even though it could make sense, since it has underlying file.
In short - I'd just continue with sync version, because there is no benefit in this case to use "async" one. Static analyzer of course doesn't know that, it only sees that there is Async-named analog of the method you use.
await sourceStream.ReadAsync(bytes, 0, bytes.Length, token).ConfigureAwait(false);
Check like this
Related
I have read this question/answer: How do I copy the contents of one stream to another?.
And I've read the doc pages of Stream.CopyToAsync and Stream.CopyTo.
I haven't found anything in any of them which either:
Tells me how to achieve this.
Confirms whether those methods do this automatically.
So it's possible that the answer is "yes, just use those methods", but that's not something that I can see documented on any of those...
I have an existing Stream object (A).
I have an async void method PopulateStream() which is going to write data into that stream.
I want to create a new Stream object (B), and I want the data from above to be written onto that Stream.
I don't want to wait for all the data to have been written to A before it starts being written to B.
i.e. I want the data to "Stream" from one Stream object to the other, as it is written.
I don't have any control over the creation of Stream A, though I do have a reference to it.
I don't have any control over the body of PopulateStream()
Without point 4. I could just await PopulateStream(), and the use the copy methods.
Can I just call the copy method before I await PopulateStream()? Will that copy future data added to Stream A? Or will it only copy the data that's currently there (which in this case would be ... nothing, yet. 😅
If I can do this, then do I need to do anything to Stream B after PopulateStream() completes, to indicate that all the data has been written? Presumably something in PopulateStream() already does that for Stream A?
A little background: I've been experimenting with using the FILE_FLAG_NO_BUFFERING flag when doing IO with large files. We're trying to reduce the load on the cache manager in the hope that with background IO, we'll reduce the impact of our app on user machines. Performance is not an issue. Being behind the scenes as much as possible is a big issue. I have a close-to-working wrapper for doing unbuffered IO but I ran into a strange issue. I get this error when I call Read with an offset that is not a multiple of 4.
Handle does not support synchronous operations. The parameters to the FileStream constructor may need to be changed to indicate that the handle was opened asynchronously (that is, it was opened explicitly for overlapped I/O).
Why does this happen? And is doesn't this message contradict itself? If I add the Asynchronous file option I get an IOException(The parameter is incorrect.)
I guess the real question is what do these requirements, http://msdn.microsoft.com/en-us/library/windows/desktop/cc644950%28v=vs.85%29.aspx, have to do with these multiples of 4.
Here is the code that demonstrates the issue:
FileOptions FileFlagNoBuffering = (FileOptions)0x20000000;
int MinSectorSize = 512;
byte[] buffer = new byte[MinSectorSize * 2];
int i = 0;
while (i < MinSectorSize)
{
try
{
using (FileStream fs = new FileStream(#"<some file>", FileMode.Open, FileAccess.Read, FileShare.None, 8, FileFlagNoBuffering | FileOptions.Asynchronous))
{
fs.Read(buffer, i, MinSectorSize);
Console.WriteLine(i);
}
}
catch { }
i++;
}
Console.ReadLine();
When using FILE_FLAG_NO_BUFFERING, the documented requirement is that the memory address for a read or write must be a multiple of the physical sector size. In your code, you've allowed the address of the byte array to be randomly chosen (hence unlikely to be a multiple of the physical sector size) and then you're adding an offset.
The behaviour you're observing is that the call works if the offset is a multiple of 4. It is likely that the byte array is aligned to a 4-byte boundary, so the call is working if the memory address is a multiple of 4.
Therefore, your question can be rewritten like this: why is the read working when the memory address is a multiple of 4, when the documentation says it has to be a multiple of 512?
The answer is that the documentation doesn't make any specific guarantees about what happens if you break the rules. It may happen that the call works anyway. It may happen that the call works anyway, but only in September on even-numbered years. It may happen that the call works anyway, but only if the memory address is a multiple of 4. (It is likely that this depends on the specific hardware and device drivers involved in the read operation. Just because it works on your machine doesn't mean it will work on anybody else's.)
It probably isn't a good idea to use FILE_FLAG_NO_BUFFERING with FileStream in the first place, because I doubt that FileStream actually guarantees that it will pass the address you give it unmodified to the underlying ReadFile call. Instead, use P/Invoke to call the underlying API functions directly. You may also need to allocate your memory this way, because I don't know whether .NET provides any way to allocate memory with a particular alignment or not.
Just call CreateFile directly with FILE_FLAG_NO_BUFFERING and then close it before opening with FileStream to achieve the same effect.
I have some code that does
MemoryStream ms = new MemoryStream();
...
return Image.FromStream(ms);
It fails in very eclectic ways since the Image object does not hold a ref to the stream, so it can get disposed if the GC kicks in which results in GDI+ errors.
How do I work around this (without saving the stream to disk, or altering my method sigs) ?
This seems highly unlikely to me - it would cause a problem for almost any use of Image.FromStream.
It seems more likely to me that something's disposing of your MemoryStream, which it shouldn't.
Could you provide a short but complete program which demonstrates the problem? Forcing garbage collection should make it relatively easy to reproduce - you could even create your own class deriving from MemoryStream with a finalizer to show whether or not it really is being collected (well, finalized at least).
There isn't a way to do it without changing your code somewhat. The Remarks section for the documentation for the static FromStream method on the Image class states:
You must keep the stream open for the
lifetime of the Image.
That being said, you have to make sure that while the Image is accessing the Stream, the stream is open. It would also appear (looking through Reflector) that the FromImage method doesn't actually cause the Image instance to hold onto a reference to the Stream the image was loaded from.
That being said, you to somehow link the image and the MemoryStream (or Stream) together so that it doesn't get GCed. If don't really retain "ownership" of the image (it is passed around), then I recommend that you create a data structure which will hold the reference to the Image and to the Stream and pass the two around in tandem.
We have an application where we need to de-serialize some data from one stream into multiple objects.
The Data array represents a number of messages of variable length packed together. There are no message delimiting codes in the stream.
We want to do something like:
void Decode(byte[] Data)
{
Object0.ExtractMessage(Data);
Object1.ExtractMessage(Data);
Object2.ExtractMessage(Data);
...
}
where each ProcessData call knows where to start in the array. Ideally we'd do this without passing a DataIx reference in.
To do this in C++ we'd just hand around a pointer into the array, and each ProcessData function would increment it as required.
Each object class knows how its own messages are serialized and can be relied upon (in C++) to return the pointer at the beginning of the next message in the stream.
Is there some inbuilt mechanism we can use to do this (without going unsafe)? The operation is high frequency (~10kps) and very lightweight. We also don't want to go copying or trimming the array.
Thanks for your help.
Could you not just pass in and return the array index? That is basically all that a pointer is anyway, an offset from a fixed memory location.
Well this sounds like you want a simple stream (E.g. just use MemoryStream as a wrapper around your byte array: stream = new MemoryStream (data)). Just wrap the byte array into a stream and every object reads as much from the stream as it needs and then hands over the stream to the next item. It even has the benefit that you aren't forced to loading the entire byte-array at once.
Other than that you can use pointers in C# exactly the way you did in C++ (though pointers require the unsafe keyword and they are discouraged)
Alternatively you could just pass data and an index variable and then increment the index (which is, in effect, the same as using a pointer but doesn't need unsafe).
How about wrapping the data in a MemoryStream and then passing a StreamReader into the ExtractMessage method?
I guess several things come to mind.
You could simulate the action of the pointer by wrapping the byte[] in a class which also maintained the array offset. Whenever you access the array you would access it thru the class, probably via an accessor method, which returned the next byte and also incremented the offset variable. The class instance could be passed between the different ExtractMessage function calls.
How about using C++/CLI? This would allow you to use familiar C/C++ techniques, and yet be directly callable from C# without the dreaded interop.
Then of course there is the dreaded unsafe option, whereby you obtain a C# pointer to the byte[] and perform the required pointer arithmetic.
You could create a stream from the byte array.
Stream stream = new MemoryStream(data);
Then your processor's could work on streams instead.
I am trying to use the EndianBinaryReader and EndianBinaryWriter that Jon Skeet wrote as part of his misc utils lib. It works great for the two uses I have made of it.
The first reading from a Network Stream (TCPClient) where I sit in a loop reading the data as it comes in. I can create a single EndianBinaryReader and then just dispose of it on the shut down of the application. I construct the EndianBinaryReader by passing the TCPClient.GetStream in.
I am now trying to do the same thing when reading from a UdpClient but this does not have a stream as it is connection less. so I get the data like so
byte[] data = udpClientSnapShot.Receive(ref endpoint);
I could put this data into a memory stream
var memoryStream = new MemoryStream(data);
and then create the EndianBinaryReader
var endianbinaryReader = new EndianBinaryReader(
new BigEndianBitConverter(), memoryStream,Encoding.ASCII);
but this means I have to create a new endian reader every time I do a read. Id there a way where I can just create a single stream that I can just keep updateing the inputstream with the data from the udp client?
I can't remember whether EndianBinaryReader buffers - you could overwrite a single MemoryStream? But to be honest there is very little overhead from an extra object here. How big are the packets? (putting it into a MemoryStream will clone the byte[]).
I'd be tempted to use the simplest thing that works and see if there is a real problem. Probably the one change I would make is to introduce using (since they are IDisposable):
using(var memoryStream = new MemoryStream(data))
using(var endianbinaryReader = ..blah..) {
// use it
}
Your best option is probably an override of the .NET Stream class to provide your custom functionality. The class is designed to be overridable with custom behavior.
It may look daunting because of the number of members, but it is easier than it looks. There are a number of boolean properties like "CanWrite", etc. Override them and have them all return "false" except for the functionality that your reader needs (probably CanRead is the only one you need to be true.)
Then, just override all of the methods that start with the phrase "When overridden in a derived class" in the help for Stream and have the unsupported methods return an "UnsupportedException" (instead of the default "NotImplementedException".
Implement the Read method to return data from your buffered UDP packets using perhaps a linked list of buffers, setting used buffers to "null" as you read past them so that the memory footprint doesn't grow unbounded.