I've been trying to use the C# MonoTorrent Library, but the lack of a documentation isn't helping. I'm trying to stream a file, but to do that, I somehow need an event whenever a Piece is written to file, or something similar.
I know there is an event which gets triggered whenever a Piece has been hashed, but it's not so useful when I need the actual content.
So I want to ask how I can know when a piece has been written to a file, so I can parse that and then stream that movie.
I already looked at the TorrentManager the ClientEngine the DiskManager and I haven't found anything useful in any of these classes nor any other Manager class. Now is this feature just hidden somewhere or do I have to do something different to get the pieces that were downloaded?
The PieceHashed event is what you need. When that event is raised you can be guaranteed that the data associated with that piece has been received, validated and written to the DiskManager.
If a MemoryWriter is being used that data may not be written to the underlying harddrive/SSD when the event is raised. To guarantee that you'll need to call the FlushAsync method, passing in the TorrentManager and piece index. If a piece is 256kB in size and there are three files of length 200kb, 50kb and 6kb contained within piece 6, all three of those files will be flushed if you pass in '6' as the piece index. If you call the overload which doesn't take a PieceIndex it will instead flush every file.
If you're writing something like a SlidingWindowPicker then you should probably only call FlushAsync when a piece in the high priority set has been downloaded. Any call to flush will flush all pending data for a given file (or all files). If the only data you have is from the very end of the torrent then flushing it immediately won't impact your ability to stream, but it may increase the overheads.
There is an alternative, which is to implement IPieceWriter and create a wrapper which flushes whenever an appropriate piece is written.
The existing MemoryWriter shows how to create a wrapper. It's Write method is implemented as follows: https://github.com/mono/monotorrent/blob/2209922c4159e394242c6c337401571312642b6e/src/MonoTorrent/MonoTorrent.Client.PieceWriters/MemoryWriter.cs#L118-L123 .
If you wanted to write something to automatically flush pieces you'd do something like:
public void Write(TorrentFile file, long offset, byte[] buffer, int bufferOffset, int count, bool forceWrite)
{
Writer.Write(file, offset, buffer, bufferOffset, count);
Writer.Flush(file);
}
Related
I have read this question/answer: How do I copy the contents of one stream to another?.
And I've read the doc pages of Stream.CopyToAsync and Stream.CopyTo.
I haven't found anything in any of them which either:
Tells me how to achieve this.
Confirms whether those methods do this automatically.
So it's possible that the answer is "yes, just use those methods", but that's not something that I can see documented on any of those...
I have an existing Stream object (A).
I have an async void method PopulateStream() which is going to write data into that stream.
I want to create a new Stream object (B), and I want the data from above to be written onto that Stream.
I don't want to wait for all the data to have been written to A before it starts being written to B.
i.e. I want the data to "Stream" from one Stream object to the other, as it is written.
I don't have any control over the creation of Stream A, though I do have a reference to it.
I don't have any control over the body of PopulateStream()
Without point 4. I could just await PopulateStream(), and the use the copy methods.
Can I just call the copy method before I await PopulateStream()? Will that copy future data added to Stream A? Or will it only copy the data that's currently there (which in this case would be ... nothing, yet. 😅
If I can do this, then do I need to do anything to Stream B after PopulateStream() completes, to indicate that all the data has been written? Presumably something in PopulateStream() already does that for Stream A?
I have been using this code to capture the webcam and I have been trying to learn from it and make it better. Rider IDE suggested I should use an async variant of MemoryMappedViewStream.Read but it doesn't work at all. It produces all-black images suggesting the async and sync methods are totally different. I am wondering why that's the case?
// Working:
sourceStream.Read(MemoryMarshal.AsBytes(image.GetPixelMemoryGroup().Single().Span));
// NOT Working:
var bytes = MemoryMarshal.AsBytes(image.GetPixelMemoryGroup().Single().Span).ToArray();
await sourceStream.ReadAsync(bytes, 0, bytes.Length, token);
Repository and line of code
Those two versions are not the same. In "sync" version you obtain a reference to memory location of an image via image.GetPixelMemoryGroup(). Then you read data from sourceStream directly into that location.
In "async" version you again obtain reference to memory location via image.GetPixelMemoryGroup but then you do something different - you call ToArray. This extension method copies bytes from image memory location into new array, the one you hold in bytes variable. You then read data from sourceStream into that bytes array, NOT directly into image memory locaiton. Then you discard bytes array, so you read them to nowhere basically.
Now,MemoryMappedViewStream inherits from UnmanagedMemoryStream and all read\write operations are implemented in UnmanagedMemoryStream. This kind of stream represents data in memory and there is nothing async it can do. The only reason it even has ReadAsync is because base stream class (Stream) has those methods. Even if you manage to make ReadAsync work - in this case it will not be asynchornous anyway. As far as I know - MemoryMappedViewStream does now allow real asynchronous access, even though it could make sense, since it has underlying file.
In short - I'd just continue with sync version, because there is no benefit in this case to use "async" one. Static analyzer of course doesn't know that, it only sees that there is Async-named analog of the method you use.
await sourceStream.ReadAsync(bytes, 0, bytes.Length, token).ConfigureAwait(false);
Check like this
A little background: I've been experimenting with using the FILE_FLAG_NO_BUFFERING flag when doing IO with large files. We're trying to reduce the load on the cache manager in the hope that with background IO, we'll reduce the impact of our app on user machines. Performance is not an issue. Being behind the scenes as much as possible is a big issue. I have a close-to-working wrapper for doing unbuffered IO but I ran into a strange issue. I get this error when I call Read with an offset that is not a multiple of 4.
Handle does not support synchronous operations. The parameters to the FileStream constructor may need to be changed to indicate that the handle was opened asynchronously (that is, it was opened explicitly for overlapped I/O).
Why does this happen? And is doesn't this message contradict itself? If I add the Asynchronous file option I get an IOException(The parameter is incorrect.)
I guess the real question is what do these requirements, http://msdn.microsoft.com/en-us/library/windows/desktop/cc644950%28v=vs.85%29.aspx, have to do with these multiples of 4.
Here is the code that demonstrates the issue:
FileOptions FileFlagNoBuffering = (FileOptions)0x20000000;
int MinSectorSize = 512;
byte[] buffer = new byte[MinSectorSize * 2];
int i = 0;
while (i < MinSectorSize)
{
try
{
using (FileStream fs = new FileStream(#"<some file>", FileMode.Open, FileAccess.Read, FileShare.None, 8, FileFlagNoBuffering | FileOptions.Asynchronous))
{
fs.Read(buffer, i, MinSectorSize);
Console.WriteLine(i);
}
}
catch { }
i++;
}
Console.ReadLine();
When using FILE_FLAG_NO_BUFFERING, the documented requirement is that the memory address for a read or write must be a multiple of the physical sector size. In your code, you've allowed the address of the byte array to be randomly chosen (hence unlikely to be a multiple of the physical sector size) and then you're adding an offset.
The behaviour you're observing is that the call works if the offset is a multiple of 4. It is likely that the byte array is aligned to a 4-byte boundary, so the call is working if the memory address is a multiple of 4.
Therefore, your question can be rewritten like this: why is the read working when the memory address is a multiple of 4, when the documentation says it has to be a multiple of 512?
The answer is that the documentation doesn't make any specific guarantees about what happens if you break the rules. It may happen that the call works anyway. It may happen that the call works anyway, but only in September on even-numbered years. It may happen that the call works anyway, but only if the memory address is a multiple of 4. (It is likely that this depends on the specific hardware and device drivers involved in the read operation. Just because it works on your machine doesn't mean it will work on anybody else's.)
It probably isn't a good idea to use FILE_FLAG_NO_BUFFERING with FileStream in the first place, because I doubt that FileStream actually guarantees that it will pass the address you give it unmodified to the underlying ReadFile call. Instead, use P/Invoke to call the underlying API functions directly. You may also need to allocate your memory this way, because I don't know whether .NET provides any way to allocate memory with a particular alignment or not.
Just call CreateFile directly with FILE_FLAG_NO_BUFFERING and then close it before opening with FileStream to achieve the same effect.
I run an algorithm that receives out-of-process messages of different types. The incoming messages are actually byte arrays and each byte array is pre-pend by a byte array flag indicating the message type. I like to understand whether it is possible to setup an IPropagator<byte[], byte[]> that processes the incoming byte arrays, interprets the byte array flags and then streams the byte array to a specific corresponding linked ActionBlock.
For example lets say I have 2 different message types and I have 2 different corresponding ActionBlocks that should only receive messages that match with the intended message type they are supposed to receive. I believe if I just link the IPropagatorBlock to both Actionblocks that both ActionBlocks will receive the same message? How can I correctly allocate each message depending on its flag (do not worry about the flag, the identification is trivial, lets assume I know at any time to which ActionBlock IPropgatorBlock wants to stream the message)? I am struggling with correctly setting up the data flow structure. I would love to be able to link the data blocks directly to each other rather than having to Post(). Is that possible?
Any help in that regards is much appreciated.
This depends on the IPropagatorBlock that you're using. If it's a custom one, you can do anything, including for example recognizing which target block to use based on the order they're linked (or something more reliable).
But assuming the block is actually a normal TransformBlock (or something like that), I think the best option is to use the overload of LinkTo() that takes a predicate, and adding the flag to the output type (which means changing the type of the block to IPropagatorBlock<byte[], Tuple<FlagType, byte[]>>, or a custom type instead of the Tuple). If you do this, then you can be sure that the target will receive the message only if the predicate for that target matches the message.
Also, what happens if you link one source block to more target blocks depends on the source block. In most cases, it sends each message to exactly one target: it first tries the first target, and only tries the second one if the first one declines or postpones the message. The exception to this rule is BroadcastBlock (and the similar WriteOnceBlock), that always tries to send each message to all targets. Again, a custom block can behave any way it wants.
I need advice on how to have my C# console application display text to the user through the standard output while still being able access it later on. The actual feature I would like to implement is to dump the entire output buffer to a text file at the end of program execution.
The workaround I use while I don't find a cleaner approach is to subclass TextWriter overriding the writing methods so they would both write to a file and call the original stdout writer. Something like this:
public class DirtyWorkaround {
private class DirtyWriter : TextWriter {
private TextWriter stdoutWriter;
private StreamWriter fileWriter;
public DirtyWriter(string path, TextWriter stdoutWriter) {
this.stdoutWriter = stdoutWriter;
this.fileWriter = new StreamWriter(path);
}
override public void Write(string s) {
stdoutWriter.Write(s);
fileWriter.Write(s);
fileWriter.Flush();
}
// Same as above for WriteLine() and WriteLine(string),
// plus whatever methods I need to override to inherit
// from TextWriter (Encoding.Get I guess).
}
public static void Main(string[] args) {
using (DirtyWriter dw = new DirtyWriter("path", Console.Out)) {
Console.SetOut(dw);
// Teh codez
}
}
}
See that it writes to and flushes the file all the time. I'd love to do it only at the end of the execution, but I couldn't find any way to access to the output buffer.
Also, excuse inaccuracies with the above code (had to write it ad hoc, sorry ;).
The perfect solution for this is to use log4net with a console appender and a file appender. There are many other appenders available as well. It also allows you to turn the different appenders off and on at runtime.
I don't think there's anything wrong with your approach.
If you wanted reusable code, consider implementing a class called MultiWriter or somesuch that takes as input two (or N?) TextWriter streams and distributes all writs, flushes, etc. to those streams. Then you can do this file/console thing, but just as easily you can split any output stream. Useful!
Probably not what you want, but just in case... Apparently, PowerShell implements a version of the venerable tee command. Which is pretty much intended for exactly this purpose. So... smoke 'em if you got 'em.
I would say mimic the diagnostics that .NET itself uses (Trace and Debug).
Create a "output" class that can have different classes that adhere to a text output interface. You report to the output class, it automatically sends the output given to the classes you have added (ConsoleOutput, TextFileOutput, WhateverOutput).. And so on.. This also leaves you open to add other "output" types (such as xml/xslt to get a nicely formatted report?).
Check out the Trace Listeners Collection to see what I mean.
Consider refactoring your application to separate the user-interaction portions from the business logic. In my experience, such a separation is quite beneficial to the structure of your program.
For the particular problem you're trying to solve here, it becomes straightforward for the user-interaction part to change its behavior from Console.WriteLine to file I/O.
I'm working on implementing a similar feature to capture output sent to the Console and save it to a log while still passing the output in real time to the normal Console so it doesn't break the application (eg. if it's a console application!).
If you're still trying to do this in your own code by saving the console output (as opposed to using a logging system to save just the information you really care about), I think you can avoid the flush after each write, as long as you also override Flush() and make sure it flushes the original stdoutWriter you saved as well as your fileWriter. You want to do this in case the application is trying to flush a partial line to the console for immediate display (such as an input prompt, a progress indicator, etc), to override the normal line-buffering.
If that approach has problems with your console output being buffered too long, you might need to make sure that WriteLine() flushes stdoutWriter (but probably doesn't need to flush fileWriter except when your Flush() override is called). But I would think that the original Console.Out (actually going to the console) would automatically flush its buffer upon a newline, so you shouldn't have to force it.
You might also want to override Close() to (flush and) close your fileWriter (and probably stdoutWriter as well), but I'm not sure if that's really needed or if a Close() in the base TextWriter would issue a Flush() (which you would already override) and you might rely on application exit to close your file. You should probably test that it gets flushed on exit, to be sure. And be aware that an abnormal exit (crash) likely won't flush buffered output. If that's an issue, flushing fileWriter on newline may be desirable, but that's another tricky can of worms to work out.