I trying to copy an incoming HTTP stream into another stream and use that instead.
When I use Request.Content.ReadAsStreamAsync().Result directly, everything works fine.
The problem that I am facing is how to "channel" the original stream thru another one.
I wrote the flowing method, but the problem is that it either blocks until all the stream data has arrived (when I use Wait()), or it returns immediately and exists without capturing any data.
private Task<Stream> GetAudioStream(RecordingSession recordingSession)
{
Task<Stream> task = Task<Stream>.Factory.StartNew(() =>
{
Request.Content.ReadAsStreamAsync().Result.CopyToAsync(recordingSession.AudioStream).Wait();
return recordingSession.AudioStream;
});
return task;
}
Perhaps, you tackle the problem from the wrong side. CopyToAsync gives you a complete copy of the stream, the Task returned by CopyToAsync is completed when all data has been copied.
If you don't need the whole complete copy of the stream, or don't want to wait for all data, then copy it manually yourself and process each chunk:
static async Task<Stream> CopyAndProcessAsync(RecordingSession recordingSession, CancellationToken token)
{
var srcStream = await Request.Content.ReadAsStreamAsync();
var dstStream = recordingSession.AudioStream;
var buffer = new byte[4096];
int bytesRead;
while ((bytesRead = await srcStream.ReadAsync(buffer, 0, buffer.Length, token).ConfigureAwait(false)) != 0)
{
await dstStream.WriteAsync(buffer, 0, bytesRead, token).ConfigureAwait(false);
// do whatever you want with the chunk
PlaybackChunk(buffer);
}
return dstStream;
}
Do this wherever you'd otherwise be consuming the result of your GetAudioStream. It well may turn out that you don't need a copy at all, when you've processed each chunk.
I think you mean something like this:
private async Task<Stream> GetAudioStream(RecordingSession recordingSession)
{
var result = await Request.Content.ReadAsStreamAsync();
await result.CopyToAsync(recordingSession.AudioStream);
return recordingSession.AudioStream;
}
async methods return Tasks. Very rarely should a non-async method return a Task.
Related
I Have endpoint which use handlers of 2 others endpoints it's probably not best practice, but it's not the point. In this methods I use a lot of MemoryStreams, ZipStream and stuff like that. Of course I dispose all of them. And everything works good till I run all tests together, then tests throw errors like: “Input string was not in a correct format.”, "Cannot read Zip file" or other weird messages. This are also test of this 2 handlers which I use in previous test.
Solution what I found is to add "Thread.Sleep(1);" at the end of the "Handle" method, just before return. It looks like something need more time to dispose, but why?. Have you any ideas why this 1ms sleep help with this?
ExtractFilesFromZipAndWriteToGivenZipArchive is an async method.
public async Task<MemoryStream> Handle(MultipleTypesExportQuery request, CancellationToken cancellationToken)
{
var stepwiseData = await HandleStepwise(request.RainmeterId, request.StepwiseQueries, cancellationToken);
var periodicData = await HandlePeriodic(request.RainmeterId, request.PeriodicQueries, cancellationToken);
var data = new List<MemoryStream>();
data.AddRange(stepwiseData);
data.AddRange(periodicData);
await using (var ms = new MemoryStream())
using (var archive = new ZipArchive(ms, ZipArchiveMode.Create,false))
{
int i = 0;
foreach (var d in data)
{
d.Open();
d.Position = 0;
var file = ZipFile.Read(d);
ExtractFilesFromZipAndWriteToGivenZipArchive(file, archive, i, cancellationToken);
i++;
file.Dispose();
d.Dispose();
}
//Thread.Sleep(100);
return ms;
}
}
ExtractFilesFromZipAndWriteToGivenZipArchive() is an asynchronous function which means, in this case, that you need to await it:
await ExtractFilesFromZipAndWriteToGivenZipArchive(file, archive, i, cancellationToken);
Otherwise, the execution will keep going without waiting the function to return.
I send a 7 byte TX command I am to receive a 7 byte RX response.
If possible, I'd like to chain the receive bytes task to send bytes task as this only seems logical.
I am aware that background workers are outdated and would like to modernize the code by utilizing the Task-Parallel-Library.
Can someone elucidate how to replace the backgroundWorker_DoWork event handler with an async task for the context described?
If you are always first sending bytes, and then receiving bytes, you could use something like this:
public static class NetworkStreamExtensions
{
public static async Task<byte[]> SendReceiveBytes(this NetworkStream self, byte[] toSend, int bytesToReceive, CancellationToken cancel)
{
await self.WriteAsync(toSend, 0, toSend.Length, cancel);
var receiveBuffer = new byte[bytesToReceive];
var receivedBytes = await self.ReadAsync(receiveBuffer, 0, receiveBuffer.Length, cancel);
if (receivedBytes != receiveBuffer.Length)
{
// Handle incorrect number of received bytes,
}
return receiveBuffer;
}
}
As this is using async/await it will use the current synchronization context to run the code. I.e. If called on the main thread, it will send and receive on the main thread. It will however permit other code to run between Writing and reading, since async methods does not block execution.
I have a HTTP proxy server which acts as a middle-man. It basically does the following:
Listen for client-browser request
Forward the request to the server
Parse the server response
Forward the response back to client-browser
So basically there is one NetworkStream, or even more often a SslStream between a client-browser and the proxy, and another one between the proxy and a server.
A requirement has arisen to also forward WebSocket traffic between a client and a server.
So now when a client-browser requests a connection upgrade to websocket, and the remote server responds with HTTP code 101, the proxy server maintains these connections in order to forward further messages from client to server and vice versa.
So after the proxy has received a message from the remote server saying it's ready to switch protocols, it needs to enter a loop where both client and server streams are polled for data, and where any received data is forwarded to the other party.
The problem
WebSocket allows both sides to send messages at any time. This is especially a problem with control messages such as ping/pong, where any side could send a ping at any time and the other side is expected to reply with a pong in a timely manner. Now consider having two instances of SslStream which don't have DataAvailable property, where the only way to read data is to call Read/ReadAsync which might not return until some data is available. Consider the following pseudo-code:
public async Task GetMessage()
{
// All these methods that we await read from the source stream
byte[] firstByte = await GetFirstByte(); // 1-byte buffer
byte[] messageLengthBytes = await GetMessageLengthBytes();
uint messageLength = GetMessageLength(messageLengthBytes);
bool isMessageMasked = DetermineIfMessageMasked(messageLengthBytes);
byte[] maskBytes;
if (isMessageMasked)
{
maskBytes = await GetMaskBytes();
}
byte[] messagePayload = await GetMessagePayload(messageLength);
// This method writes to the destination stream
await ComposeAndForwardMessageToOtherParty(firstByte, messageLengthBytes, maskBytes, messagePayload);
}
The above pseudo code reads from one stream and writes to the other. The problem is that the above procedure needs to be run for both streams simultaneously, because we don't know which side would send a message to the other at any given point in time. However, it is impossible to perform a write operation while there is a read operation active. And because we don't have the means necessary to poll for incoming data, read operations have to be blocking. That means if we start read operations for both streams at the same time, we can forget about writing to them. One stream will eventually return some data, but we won't be able to send that data to the other stream as it will still be busy trying to read. And that might take a while, at least until the side that owns that stream sends a ping request.
Thanks to comments from #MarcGravell we've learned that independent read/write operations are supported with network streams, i.e. NetworkStream acts as two independent pipes - one read, one write - it is fully duplex.
Therefore, proxying WebSocket messages can be as easy as just starting two independent tasks, one to read from client stream and write to server stream, and another to read from server stream and write to client stream.
If it can be of any help to anyone searching for it, here is how I implemented that:
public class WebSocketRequestHandler
{
private const int MaxMessageLength = 0x7FFFFFFF;
private const byte LengthBitMask = 0x7F;
private const byte MaskBitMask = 0x80;
private delegate Task WriteStreamAsyncDelegate(byte[] buffer, int offset, int count, CancellationToken cancellationToken);
private delegate Task<byte[]> BufferStreamAsyncDelegate(int count, CancellationToken cancellationToken);
public async Task HandleWebSocketMessagesAsync(CancellationToken cancellationToken = default(CancellationToken))
{
var clientListener = ListenForClientMessages(cancellationToken);
var serverListener = ListenForServerMessages(cancellationToken);
await Task.WhenAll(clientListener, serverListener);
}
private async Task ListenForClientMessages(CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
cancellationToken.ThrowIfCancellationRequested();
await ListenForMessages(YOUR_CLIENT_STREAM_BUFFER_METHOD_DELEGATE, YOUR_SERVER_STREAM_WRITE_METHOD_DELEGATE, cancellationToken);
}
}
private async Task ListenForServerMessages(CancellationToken cancellationToken)
{
while (!cancellationToken.IsCancellationRequested)
{
cancellationToken.ThrowIfCancellationRequested();
await ListenForMessages(YOUR_SERVER_STREAM_BUFFER_METHOD_DELEGATE, YOUR_CLIENT_STREAM_WRITE_METHOD_DELEGATE, cancellationToken);
}
}
private static async Task ListenForMessages(BufferStreamAsyncDelegate sourceStreamReader,
WriteStreamAsyncDelegate destinationStreamWriter,
CancellationToken cancellationToken)
{
var messageBuilder = new List<byte>();
var firstByte = await sourceStreamReader(1, cancellationToken);
messageBuilder.AddRange(firstByte);
var lengthBytes = await GetLengthBytes(sourceStreamReader, cancellationToken);
messageBuilder.AddRange(lengthBytes);
var isMaskBitSet = (lengthBytes[0] & MaskBitMask) != 0;
var length = GetMessageLength(lengthBytes);
if (isMaskBitSet)
{
var maskBytes = await sourceStreamReader(4, cancellationToken);
messageBuilder.AddRange(maskBytes);
}
var messagePayloadBytes = await sourceStreamReader(length, cancellationToken);
messageBuilder.AddRange(messagePayloadBytes);
await destinationStreamWriter(messageBuilder.ToArray(), 0, messageBuilder.Count, cancellationToken);
}
private static async Task<byte[]> GetLengthBytes(BufferStreamAsyncDelegate sourceStreamReader, CancellationToken cancellationToken)
{
var lengthBytes = new List<byte>();
var firstLengthByte = await sourceStreamReader(1, cancellationToken);
lengthBytes.AddRange(firstLengthByte);
var lengthByteValue = firstLengthByte[0] & LengthBitMask;
if (lengthByteValue <= 125)
{
return lengthBytes.ToArray();
}
switch (lengthByteValue)
{
case 126:
{
var secondLengthBytes = await sourceStreamReader(2, cancellationToken);
lengthBytes.AddRange(secondLengthBytes);
return lengthBytes.ToArray();
}
case 127:
{
var secondLengthBytes = await sourceStreamReader(8, cancellationToken);
lengthBytes.AddRange(secondLengthBytes);
return lengthBytes.ToArray();
}
default:
throw new Exception($"Unexpected first length byte value: {lengthByteValue}");
}
}
private static int GetMessageLength(byte[] lengthBytes)
{
byte[] subArray;
switch (lengthBytes.Length)
{
case 1:
return lengthBytes[0] & LengthBitMask;
case 3:
if (!BitConverter.IsLittleEndian)
{
return BitConverter.ToUInt16(lengthBytes, 1);
}
subArray = lengthBytes.SubArray(1, 2);
Array.Reverse(subArray);
return BitConverter.ToUInt16(subArray, 0);
case 9:
subArray = lengthBytes.SubArray(1, 8);
Array.Reverse(subArray);
var retVal = BitConverter.ToUInt64(subArray, 0);
if (retVal > MaxMessageLength)
{
throw new Exception($"Unexpected payload length: {retVal}");
}
return (int) retVal;
default:
throw new Exception($"Impossibru!!1 The length of lengthBytes array was: '{lengthBytes.Length}'");
}
}
}
It can be used by just calling await handler.HandleWebSocketMessagesAsync(cancellationToken) after the initial handshake has been performed.
The SubArray method is taken from here: https://stackoverflow.com/a/943650/828023 (also from #Marc haha)
I'm trying to learn the async and await mechanisms in C#.
The simplest example is clear to me.
The line
Task<string> getStringTask = client.GetStringAsync("http://msdn.microsoft.com");
triggers an asynchronous web call. The control returns to AccessTheWebAsync(). It is free to perform DoIndependentWork(). After doing this it waits for the completion of the task getStringTask and when this result is available the function executes the next line
return urlContents.Length;
So, as far as I understand the purpose of the async call is to let the caller execute other operations when the operation tagged with async is in progress.
However, I'm bit confused with the example, in this function.
private async Task<byte[]> GetURLContentsAsync(string url)
{
// The downloaded resource ends up in the variable named content.
var content = new MemoryStream();
// Initialize an HttpWebRequest for the current URL.
var webReq = (HttpWebRequest)WebRequest.Create(url);
// Send the request to the Internet resource and wait for
// the response.
using (WebResponse response = await webReq.GetResponseAsync())
// The previous statement abbreviates the following two statements.
//Task<WebResponse> responseTask = webReq.GetResponseAsync();
//using (WebResponse response = await responseTask)
{
// Get the data stream that is associated with the specified url.
using (Stream responseStream = response.GetResponseStream())
{
// Read the bytes in responseStream and copy them to content.
await responseStream.CopyToAsync(content);
// The previous statement abbreviates the following two statements.
// CopyToAsync returns a Task, not a Task<T>.
//Task copyTask = responseStream.CopyToAsync(content);
// When copyTask is completed, content contains a copy of
// responseStream.
//await copyTask;
}
}
// Return the result as a byte array.
return content.ToArray();
}
Inside the method GetURLContentsAsync(), there are two async invocations. However, the API waits with an await call on both. The caller is not doing anything between the trigger of the async operation and the receipt of the data. So, as far as I understand, the async/await mechanism brings no benefit here. Am I missing something obvious here?
Your code doesn't need to explicitly be doing anything between await'd async calls to gain benefit. It means that the thread isn't sitting waiting for each call to complete, it is available to do other work.
If this is a web application it can result in more requests being processed. If it is a Windows application it means the UI thread isn't blocked and the user has a better experience.
However, the API waits with an await call on both.
You will have to await for the both because your method code should get executed sequentially, if you don't await the first call, your next lines of code will also get executed which is something you might not expect or need to happen.
The following two reasons that come in my mind for awaiting both methods are:
it is possible that your first async method result is used as
parameter in your second async method call
it is also possible that we decide on the result of first async
method call that the second async method to be called or not
So if that's the case then it is quite clear why you would not need to add await to every async method call inside your async method.
EDIT:
From the example which you are pointing to clearly you can see that the output of first async method is being used in the second async method call here:
using (WebResponse response = await webReq.GetResponseAsync())
// The previous statement abbreviates the following two statements.
//using (WebResponse response = await responseTask)
{
// Get the data stream that is associated with the specified url.
using (Stream responseStream = response.GetResponseStream())
{
// Read the bytes in responseStream and copy them to content.
await responseStream.CopyToAsync(content);
// The previous statement abbreviates the following two statements.
// CopyToAsync returns a Task, not a Task<T>.
//Task copyTask = responseStream.CopyToAsync(content);
// When copyTask is completed, content contains a copy of
// responseStream.
//await copyTask;
}
}
GetResponseAsync returns when the web server starts its response (by sending the headers), while CopyToAsync returns once all the data has been sent from the server and copied to the other stream.
If you add code to record how much time elapses between the start of the asynchronous call and the return to your function, you'll see that both methods take some time to complete (on a large file, at least.)
private static async Task<byte[]> GetURLContentsAsync(string url) {
var content = new MemoryStream();
var webReq = (HttpWebRequest)WebRequest.Create(url);
DateTime responseStart = DateTime.Now;
using (WebResponse response = await webReq.GetResponseAsync()) {
Console.WriteLine($"GetResponseAsync time: {(DateTime.Now - responseStart).TotalSeconds}");
using (Stream responseStream = response.GetResponseStream()) {
DateTime copyStart = DateTime.Now;
await responseStream.CopyToAsync(content);
Console.WriteLine($"CopyToAsync time: {(DateTime.Now - copyStart).TotalSeconds}");
}
}
return content.ToArray();
}
For a ~40 MB file on a fast server, the first await is quick while the second await takes longer.
https://ftp.mozilla.org/pub/thunderbird/releases/52.2.1/win32/en-US/Thunderbird%20Setup%2052.2.1.exe
GetResponseAsync time: 0.3422409
CopyToAsync time: 5.3175731
But for a server that takes a while to respond, the first await can take a while too.
http://www.fakeresponse.com/api/?sleep=3
GetResponseAsync time: 3.3125195
CopyToAsync time: 0
Currently I am implementing a way to report Progress with the HttpClient, since we share code with a .NET4 WPF and a Windows Universal App we use the Microsoft HTTP Client Libraries from NuGet. The idea was to wrap the target file stream in a CountingInputStream and report progress there:
public override void Write(byte[] buffer, int offset, int count)
{
_stream.Write(buffer, offset, count);
_bytesRead += count;
_progress.Report(_bytesRead);
if (_cancellationToken.IsCancellationRequested)
{
_cancellationToken.ThrowIfCancellationRequested();
}
}
Then I send my request with: HttpResponseMessage httpResponseMessage = AsyncHelpers.RunSync(() => _httpClient.SendAsync(httpRequestMessage, HttpCompletionOption.ResponseHeadersRead, cancellationToken));
After that I open the file stream and then copy the content stream. The response has correct headers:
Content-Length: 213334
Content-Type: application/octet-stream; charset=UTF-8
Content-Disposition: attachment; filename="Bondi Beach.jpg"; filename*=UTF-8''Bondi%20Beach.jpg
using(Stream fileStream = new CountingInputStream(storage.Open(downloadRequest.TargetPath, FileMode.Create), downloadRequest.Progress, cancellationToken )) {
await HttpHeaderResponseMessage.Content.CopyToAsync(fileStream);
}
The problem is that the StreamContent only starts writing to the file stream after the download has finished. When it started writing progress reporting just works fine.
I already tried different approaches like:
ReadAsStreamAsync and then copy the response stream to file stream
ReadAsStreamAsync manually read to buffer and then write to file stream
_httpClient = new System.Net.Http.HttpClient(){MaxResponseContentBufferSize = 4096};to restrict the BufferSize
Any ideas how I could force the ContentStream to write to the file stream while it is still downloading?
UPDATE:
Following Luaans advice I tried to override the WriteAsync and implemented a StreamContent Extensions method:
//CountingInputStream
public Task WriteAsync(byte[] buffer, int offset, int count, CancellationToken cancellationToken)
{
_bytesRead += count;
_progress.Report(_bytesRead);
if (_cancellationToken.IsCancellationRequested)
{
_cancellationToken.ThrowIfCancellationRequested();
}
return _stream.WriteAsync(buffer, offset, count, cancellationToken);
}
//static Extensions Class
public static async Task CopyToAs(this StreamContent source, Stream targetStream)
{
int read;
byte[] buffer = new byte[4096];
using(Stream responseStream = await source.ReadAsStreamAsync()) {
while ((read = await responseStream.ReadAsync(buffer,0,buffer.Length))>0) {
await targetStream.WriteAsync(buffer, 0, read);
}
}
}
It still waits till the download is finish until it calls ReadAsync the first time. Any hints what I have done wrong?
The fact that ReadAsStreamAsync is, well, async, makes this rather suspicious. Why would you asynchronously wait to get the stream? You're supposed to read it asynchronously, but you should have the stream itself ready right away.
Reading the documentation makes this blatantly obvious:
This operation will not block. The returned task object will complete after the whole response (including content) is read.
However, there's overloads you can use to have it return after the headers have been read. This still means you need to wait for the server to process the request, before starting to get progress, but for the download itself, you are in luck.
Sample code:
var response =
await
(
new HttpClient()
.GetAsync("http://www.microsoft.com/", HttpCompletionOption.ResponseHeadersRead)
);
var stream = await response.Content.ReadAsStreamAsync();
var buffer = new byte[2048];
while (await stream.ReadAsync(buffer, 0, buffer.Length) > 0)
{
// Report progress and write to a different stream
}
EDIT:
It sounds like you should be using Windows.Web.Http.HttpClient instead of System.Net.Http.HttpClient:
async Task DownloadWithProgress()
{
var awaitable = httpClient.GetAsync(yourUrl)
awaitable.Progress = (res, progress) =>
{
// Report progress
}
await awaitable;
}